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Abstract 


This  study  examined  how  manipulating  the  level  of  environmental  complexity  and  the 
type  of  team  training  given  to  subject  volunteers  impacted  important  team  process  behaviors 
and  performance  outcomes.  Complexity  levels  were  manipulated  by  directly  altering  critical 
target  object  features  and  overall  planning  information  ambiguity.  This  study  also  examined 
how  subject  teams  performed  in  a  dynamic  environment  that  not  only  required  cooperative 
behavior  between  individuals  within  a  single  team,  but  also  collective  actions  between 
separate  teams  to  achieve  assigned  performance  goals.  This  effort  focused  on  testing  critical 
aspects  of  an  emerging  framework  of  multi-team  systems  (MTSs). 

The  results  from  this  research  initially  failed  to  support  the  contention  that 
manipulating  environmental  complexity  and  team  training  strongly  influenced  MTS  process 
behaviors.  However,  when  continuous  scores  were  employed  as  measures  of  the  MTS's 
collective  training  knowledge,  analysis  revealed  that  coordination  training  impacted  action 
process  behaviors  and  thereby  indirectly  improved  performance.  These  results  also  provided 
support  for  reconceptualizing  the  relationships  existing  between  central  components  of  the 
MTS  framework.  Based  on  these  findings,  transition  process  behaviors  provide  a  critical  link 
between  interpersonal  and  action  processes  in  an  MTS.  These  findings  support  the  emerging 
theory  of  MTS  and  offer  some  initial  clarifications  of  how  these  team  processes  interrelate  to 
enhance  overall  performance.  Prospective  directions  for  future  research  are  suggested. 
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Chapter  1 

INTRODUCTION 

Organizational  managers  and  researchers  (e.g.,  Salas,  Dickinson,  Converse,  & 
Tannenbaum,  1992;  Weaver,  Bowers,  Salas,  &  Cannon-Bowers,  1995)  have  long  known  that 
the  increasing  volume  of  and  reliance  on  technology  in  the  workplace  has  significantly 
contributed  to  the  complexity  of  the  employees’  performance  environment.  This  situation 
has  made  it  virtually  impossible  for  individual  employees  or  institutionalized  organizational 
structures  (e.g.,  departments,  units,  etc.)  to  complete  their  work  entirely  independent  from  the 
actions  of  other  individuals  and  entities  within  the  organization  (Salas  et  al.,  1992;  Weaver  et 
al.,  1995).  In  response  to  technological  advances  and  an  increasing  emphasis  on  collective 
outcomes  and  products,  many  organizations  have  adopted  a  team  approach  to  the  work 
environment  (Brannick,  Roach,  &  Salas,  1993;  Cannon-Bowers,  Oser,  &  Flanagan,  1992). 

Teams  are  viewed  as  more  suitable  for  complex  tasks  because  they  allow  members  to 
share  the  workload,  monitor  work  behaviors  of  other  team  members,  and  develop  and 
contribute  personal  expertise  to  various  subtasks.  This  increased  reliance  on  teams  being 
responsible  for  completing  complex  tasks  can  easily  be  seen  in  many  civilian  and  military 
contexts.  For  example,  organizations  commonly  use  top  management  teams  to  make 
important  decisions  impacting  the  operation  and  future  development  of  an  organization.  The 
military  has  also  applied  research  on  team  process  and  performance  to  such  critical 
operational  areas  as  tactical  decision  making  (e.g.,  Cannon-Bowers,  Salas,  &  Baker,  1991), 
tank  crew  training  (e.g.,  Cavanagh  &  Williams,  1987;  Tziner  &  Eden,  1985),  aircrew  training 
(e.g.,  Cavanagh  &  Williams,  1987;  Foushee,  1987)  and  aircraft  command  and  control  (e.g., 
Elliott,  Neville,  &  Dalrymple,  1996). 
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Team  performance  effectiveness  has  received  renewed  interest  in  research  and 
organizational  literature  due  to  the  pervasive  effects  of  organizational  downsizing  and 
technology  increasing  the  levels  of  complexity  and  interdependence  existing  in  the 
workplace  (Swezey  &  Salas,  1992).  In  response  to  these  developments,  leaders  and 
managers  frequently  turn  to  teams  to  effectively  deal  with  the  increasingly  complex  work 
environment.  Cannon-Bowers  et  al.  (1992)  asserted  that  teams  are  increasingly  important  in 
industry  because:  (1)  there  are  critical  organizational  tasks  which  no  single  individual  can 
accomplish  alone,  (2)  organizational  leaders  and  managers  believe  that  groups  perform  better 
than  individuals;  and  (3)  group  structures  have  emerged  in  response  to  the  growing 
humanistic  movement  in  industry.  The  increasing  importance  of  teams  in  the  workplace 
provides  a  basis  for  studying  teams  as  separate  units  of  measurement  that  can  and  should  be 
differentiated  from  individual  and  organizational  levels  of  analysis. 

This  study  examined  team  performance  in  contexts  that  have  been  largely  ignored  by 
much  of  the  empirical  research  in  the  area.  This  study  built  on  the  existing  body  of  team 
research  employing  low-fidelity  personal  computer  (PC)  flight  simulations  to  study  team 
behaviors,  team  processes,  and  performance  effectiveness  (e.g.,  Goodwin,  1997;  Mathieu  & 
Goodwin,  1997;  Stout,  Cannon-Bowers,  Salas,  &  Morgan,  1990;  Kozlowski  &  Gully,  1996). 
Specifically,  this  study  explored  how  manipulating  the  level  of  environmental  complexity 
impacted  important  team  process  activities  and  performance  outcomes  in  environments 
where  multiple  teams  must  cooperate  to  successfully  achieve  their  assigned  goals  and 
objectives. 

Rather  than  altering  the  degree  of  interdependent  activity  demanded  of  subject  teams 
or  the  ambiguity  of  task  and  environmental  cues,  this  study  varied  the  actual  level  of 
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complexity  existing  in  the  task  environment  by  directly  manipulating  critical  target  object 
features  and  information  ambiguity.  Following  the  recommendations  of  researchers 
emphasizing  that  teams  must  be  studied  in  more  naturalistic  decision  environments  (e.g., 
Kozlowski  &  Gully,  1996;  Urban,  Weaver,  Bowers,  &  Rhodenizer,  1996),  this  study 
advanced  current  team  research  findings  by  focusing  on  team  level  variables  allowing 
interdependent  teams  to  perform  in  multi-team  environments. 

Previous  researchers  examining  team  processes  and  performance  have  generally 
manipulated  environmental  complexity  by  altering  the  ambiguity  of  important  information 
cues  associated  with  higher  levels  of  threat  (e.g.,  Elliott  et  al.,  1996;  Hollenbeck  et  al.,  1997) 
or  embedded  task  events  that  deviated  from  previously  established  routines  (e.g.,  Stout, 

Salas,  &  Fowlkes,  1997)  or  treated  complexity  as  though  it  was  a  nearly  constant 
characteristic  of  the  team  task  environment  (e.g.,  McIntyre  &  Salas,  1995).  The  complexity 
of  the  environment  is  an  especially  important  consideration  in  military  combat  operations, 
since  the  consequences  of  decision  errors  due  to  misidentification  of  objects  in  the  team’s 
conceptual  field  can  be  devastating  in  terms  of  international  relations,  battle  outcomes,  and 
loss  of  lives. 

Previous  researchers  have  demonstrated  that  training  is  an  essential  ingredient  for  any 
team’s  success  in  many  different  task  environments.  In  their  extensive  review  of  team 
training,  Salas,  Dickinson,  Converse,  and  Tannenbaum  (1992)  pointed  out  that  team  training 
has  most  often  emphasized  the  individual  mastery  of  skills  within  a  team  setting.  They 
emphasized  that  there  are  many  different  variables  to  consider  when  designing  team  training 
programs.  They  included  such  variables  as  the  nature  of  the  task,  the  degree  of  coordination 
required  of  team  members,  the  fidelity  of  the  training  environment,  and  the  nature  of 
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performance  feedback  in  their  discussion  of  training  designs  and  taxonomies.  They  and  other 
researchers  (e.g.,  Funke,  1991;  Kaempf,  Klein,  Thordsen,  &  Wolf,  1996)  have  emphasized 
the  importance  of  examining  the  effectiveness  of  various  training  interventions  in  differing 
task  environments.  To  this  end,  this  study  also  examined  how  varying  the  levels  of 
environmental  complexity  affected  subject  teams’  process  behaviors  and  interacted  with 
multiple  team  training  conditions. 

Finally,  as  mentioned  earlier,  this  study  expanded  the  existing  literature  by  examining 
how  subject  teams  performed  in  a  dynamic  environment  that  not  only  required  cooperative 
behavior  between  individuals  within  a  single  team,  but  also  collective  actions  between 
separate  teams  to  achieve  assigned  performance  goals.  This  study  contributed  to  the  existing 
team  performance  literature  by  analyzing  factors  and  processes  influencing  combat  aircrew 
performance  under  psychologically  realistic  simulation  conditions,  task  demands,  and 
operational  performance  standards. 

In  Chapter  2, 1  review  the  existing  literature  and  research  examining  team  process  and 
performance  relevant  to  this  study  and  present  the  hypotheses  tested  in  this  study.  In  Chapter 
3, 1  discuss  the  experimental  methods  and  designs  used  to  test  these  hypotheses.  Chapter  4 
reviews  the  team  performance  and  process  measures  collected  for  the  analysis.  The  results  of 
the  study  and  analysis  are  presented  in  Chapter  5.  Chapter  6  provides  a  discussion  of  the 
results  and  analysis.  Finally,  Chapter  7  discusses  the  implications  of  this  study  for  the  field 
and  future  research. 
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Chapter  2 

RESEARCH  REVIEW 
General  Theoretical  Framework. 

Numerous  definitions  of  “teams”  exist  in  the  literature  (e.g.  Brannick  et  al.,  1993; 
Morgan,  Glickman,  Woodward,  Blaiwes,  &  Salas,  1986)  that  focus  on  different  attributes  of 
teams  and  groups  existing  in  various  work  environments.  I  believe  Salas,  Dickinson, 
Converse,  and  Tannenbaum’s  (1992)  definition  best  describes  the  distinguishing  features  of 
teams  commonly  found  in  large  organizations,  especially  operational  military  units.  They 
defined  a  team  as  “. .  .a  distinguishable  set  of  two  or  more  people  who  interact  dynamically, 
interdependently,  and  adaptively  toward  a  common  and  valued  goal/objective/mission,  who 
have  each  been  assigned  specific  roles  or  functions  to  perform,  and  who  have  a  limited  life¬ 
span  of  membership”  (p.4).  Central  to  this  definition  is  that  effective  performance  requires  a 
dynamic  exchange  of  information  and  resources  among  the  team  members,  coordination  of 
task  activities,  constant  adjustments  to  task  demands,  and  organizational  structuring  of  team 
members  (Salas  et  al.,  1992). 

Based  on  a  review  of  the  literature  comparing  the  two  classifications  (see  Hollenbeck 
et  al.,  1997;  Salas  et  al.,  1992;  Tannenbaum,  Salas,  &  Cannon-Bowers,  1996),  teams  can  be 
distinguished  from  groups  by  the  fact  that  all  team  members  have  some  form  of 
interdependent  task  responsibilities  and  share,  to  some  degree,  a  commonly  held  perception 
of  the  team’s  purpose  and  goals.  Hollenbeck  et  al.  (1997)  pointed  out  that,  unlike  many 
types  of  groups,  team  successes  and  failures  have  direct  consequences  for  every  team 
member.  Based  on  this  literature,  I  contend  that  teams  represent  a  more  specific 
configuration  of  individuals  than  what  are  commonly  considered  to  be  more  loosely  defined 
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groups.  Groups  tend  to  come  together  for  much  longer  periods  and  have  more  long-term, 
general  goals  than  teams.  Teams  are  associated  with  more  specific,  focused  tasks  and 
objectives  and  generally  exist  for  a  much  shorter  period  of  time  than  groups.  Since  there  is 
much  otherwise  insightful  research  that  has  treated  “groups”  (e.g.,  small  group  behavior, 
work  group  performance,  military  and  civilian  aircrews,  etc.)  and  “teams”  as  equivalent  and 
interchangeable  terms.  I’ll  honor  the  authors’  original  designations  in  any  direct  quotations 
from  specific  reports  and  publications.  Otherwise,  I’ll  use  the  term  “teams”  in  all  cases  where 
Salas  et  al.’s  (1992)  definition  applies. 

The  I-P-0  Framework. 

In  general,  most  current  research  examining  team  performance  and  effectiveness  has 
followed  Hackman  and  his  colleagues’  input-process-outcome  (I-P-O)  framework,  as 
illustrated  in  Figure  1  (see  Hackman  &  Oldham,  1980;  Hackman,  1992;  Heffner,  1996; 

Guzzo  &  Dickson,  1996). 

Figure  1:  Team  Process  Framework 


PROCESS 

-  decision  making 

-  situational  awareness 

-  communication 

-  coordination 


OUTCOME 

-  individual 
-team 

-  organization 


Inputs  include  conditions  that  exist  prior  to  a  performance  episode,  such  as  team 
composition  (i.e.,  team  member  demographics  and  characteristics),  task  structures  and 
demands,  contextual  or  situational  characteristics,  and  external  organizational  factors 


impacting  the  nature  of  the  work  environment  in  which  teams  find  themselves  (Salas  et  al., 
1992).  Performance  episodes  are  "distinguishable  periods  of  time  over  which  performance 
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accrues  and  is  reviewed"  (Mathieu  &  Button,  1992,  p.  1759).  Heffner  (1996)  pointed  out  that 
training  is  an  important  precursor  to  the  inputs  of  the  first  performance  episode  and  provides 
teams  with  a  basis  for  understanding  and  interpreting  their  performance  inputs.  An  inherent 
assumption  in  the  I-P-0  model  is  that  there  are  separate  inputs,  processes,  and  outcomes  for 
each  performance  episode.  Hackman  (1992)  pointed  out  that  individual  members  of  a  team 
also  provide  additional,  important  contextual  cues  and  inputs  for  each  member’s  behavior 
within  the  team. 

One  could  take  Hackman’s  (1992)  perspective  a  step  further  by  also  asserting  that 
teams  working  together  or  separately  in  the  same  environment  significantly  influence  each 
other’s  performance  and  behavior  (Swezey  &  Salas,  1992).  Teams  rarely  operate  totally  on 
their  own,  independent  of  the  actions  of  other  teams  and  individuals.  Even  teams  working  on 
their  own  depend  on  others  for  support  to  achieve  their  assigned  objectives.  In  many 
situations,  action  teams  are  comprised  of  smaller  teams  working  collectively  toward  some 
specified  goal.  An  example  of  this  type  of  team  can  be  seen  in  many  military  settings.  A 
typical  fighter  aircraft  crew  is  comprised  of  two  individuals  working  together.  Yet,  single 
aircraft  are  almost  never  used  in  isolation  in  combat  environments  to  achieve  significant 
mission  goals  or  objectives.  Multiple  flight  teams  are  brought  together  and  must  work  in 
concert  with  each  other  to  achieve  their  mission  objectives  and  goals.  Even  in  business, 
process  action  teams  usually  comprise  multiple  teams  working  together  and  sharing 
resources  and  task  responsibilities  to  reach  a  common  goal  (see  Peters,  1994).  Teams  almost 
always  operate  in  multi-team  environments,  yet  there  seems  to  have  been  very  little  attention 
paid  to  this  aspect  of  the  team  performance  environment. 
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In  their  review  of  networked  computer  simulations  designed  to  explore  team 
performance,  Weaver  et  al.  (1995)  noted  that  teams  are  frequently  hierarchical  and  have 
varying  degrees  of  responsibilities  and  expertise  among  their  members.  These  factors 
influence  team  member  behaviors  and  performance  by  creating  preconceived  expectations 
about  decision  making,  power  relationships,  role  responsibilities,  and  communication  flow 
within  the  team  (see  Salas  et  al.,  1992;  Weaver  et  al,  1995). 

This  is  especially  true  in  military  organizations  where  the  rank,  denoting  an 
individual’s  level  of  authority  and  position  within  the  organization,  and  experience  level  of 
individual  crew  members  may  be  very  different  within  the  same  team.  A  Second  Lieutenant, 
fresh  out  of  college  and  technical  training  school,  could  be  suddenly  put  in  charge  of  a  crew 
directing  Master  Sergeants  with  nearly  twenty  years  of  task  experience. 1  A  junior  officer 
could  also  find  himself/herself  acting  as  the  mission  commander  for  an  air  strike  against 
enemy  targets,  even  though  more  senior  officers  are  present  within  the  strike  team.  In 
deciding  whom  to  place  in  a  leadership  position  for  a  tactical  (i.e.,  battlefield  or  combat) 
military  operation,  technical  expertise  and  recent  operational  experience  are  often  more 
important  than  the  discrepancy  between  an  individual’s  rank  and  that  of  other  team  members. 
This  type  of  distributed  composition  requires  a  team  to  establish  a  balance  between  official 
positions  of  authority  and  sources  of  task  expertise  critical  to  achieving  mission  requirements 
and  desired  performance  outcomes. 

Process  factors  describe  how  team  members  interact  and  collectively  deal  with  their 
environment.  They  include  variables  such  as  communication,  cooperation,  coordination, 
cohesion,  leadership,  and  decision  making  (cf.,  Cobb,  1998;  Foushee,  1987;  Heffner,  1996; 
Salas  et  al.,  1992;  Stout  et  al.,  1997;  Tziner  &  Eden,  1985).  Process  mediates  a  team’s 
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effectiveness  in  creating  desired  outcomes  from  various  inputs.  Even  under  optimal 
environmental  conditions,  poor  team  process  can  lead  to  process  loss  (Steiner,  1972)  that 
results  in  less  than  optimal  performance.  Conversely,  in  suboptimal  operating  conditions, 
good  team  process  behaviors  may  enable  a  higher  team  performance  outcome  than  might  be 
expected  considering  the  constraints  existing  in  the  situation. 

Researchers  interested  in  examining  team  process  variables  are  immediately 
confronted  with  a  significant  conceptual  problem.  As  Cannon-Bowers,  Tannenbaum,  Salas, 
and  Volpe  (1995)  pointed  out,  there  are  numerous  inconsistencies  in  the  manner  previous 
researchers  have  labeled,  defined,  and  operationalized  many  of  the  variables  identified  as 
composing  team  process.  These  inconsistencies  potentially  represent  significant  hurdles  to 
understanding  the  results  or  previous  studies,  providing  the  conceptual  foundation  needed  to 
compare  the  results  of  various  empirical  studies,  and  generalizing  research  findings  to  more 
applied  settings.  In  response  to  these  problems,  researchers  have  relied  on  carefully  defined 
and  bounded  taxonomies  to  conceptualize  their  hypotheses  and  results. 

Fleishman  and  Zaccaro  (1992)  and  Brannick  and  his  colleagues  (Brannick  et  al, 
1993;  Prince,  Brannick,  Prince,  &  Salas,  1997)  have  created  team  process  taxonomies  that 
have  provided  a  common  conceptual  foundation  for  many  team  research  efforts.  Fleishman 
and  Zaccaro's  (1992)  taxonomy  attempted  to  classify  various  task  performance  functions. 
Their  taxonomy  includes  several  dimensions  (i.e.,  team  orientation,  resource  distribution, 
timing,  response  coordination,  motivation,  systems  monitoring,  and  procedural  maintenance) 
describing  "a  set  of  responses,  independent  and  synchronized,  that  is  separate  from  the  task 
itself'  (Fleishman  &  Zaccaro,  1992,  p.  34).  Although  it  appears  that  Fleishman  and  Zaccaro 
successfully  identified  the  task-related  and  coordination  variables  necessary  for  high 
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performance  teams,  their  dimensions  have  been  criticized  as  too  specific  to  adequately 
capture  the  varied  interactions  occurring  between  team  members  performing  their  tasks  (see 
Heffner,  1996). 

A  second  taxonomy  was  developed  and  empirically  tested  by  Brannick  and  his 
colleagues  (Brannick  et  ah,  1996;  Prince  et  ah,  1997).  Like  Fleishman  and  Zaccaro’s  (1992) 
concept,  Brannick’ s  taxonomy  also  identified  six  dimensions  composing  what  is  generally 
considered  team  process.  These  dimensions  included  leadership,  assertiveness,  decision 
making/mission  analysis,  adaptability/flexibility,  situational  awareness,  and  communication. 
Unlike  the  previous  taxonomies,  these  dimensions  were  specifically  focused  on  specific  types 
of  team  member  behaviors.  While  a  step  forward  in  many  regards,  especially  in  directing 
attention  to  team  member  actions  and  behaviors,  Brannick’s  taxonomy  repeats  one 
conceptual  labeling  error  found  in  previous  systems.  It  could  be  argued  that  many  of  the 
dimensions  Brannick  specified  are  actually  better  seen  as  emerging  team  or  team  member 
characteristics  than  actual  team  processes. 

Process  implies  that  something  is  acted  upon  to  achieve  some  defined  result. 
Leadership,  assertiveness,  situational  awareness,  and  other  labels  seem  to  define  states  or 
categories  of  team  member  behavior  rather  than  the  actual  behaviors  enacted  by  team 
members,  jointly  or  individually,  to  convert  various  team  inputs  to  desired  team  outcomes. 

In  their  description  and  examination  of  teamwork  competencies  essential  to  effective  team 
performance,  Cannon-Bowers  et  ah  (1995)  added  performance  monitoring  and  feedback,  and 
interpersonal  skills  to  Brannick’s  six  dimensions.  While  the  added  dimensions  were 
certainly  more  descriptive  of  process  related  behaviors,  the  resulting  taxonomy  seemed  too 
limited  to  comprehensively  capture  the  underlying  nature  of  team  processes. 
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In  an  effort  to  better  conceptualize  team  process  variables,  Marks,  Mathieu,  & 
Zaccaro  (in  preparation)  have  created  a  synthesis  of  existing  process  taxonomies  that  focuses 
on  the  dynamic  performance  environment  found  in  military  settings.  In  their  proposed 
taxonomy,  team  process  comprises  transition  processes,  action  processes,  and  interpersonal 
processes.  According  to  this  framework,  experimenters  should  closely  examine  the  process 
behaviors  that  teams  employ  to  convert  situational  inputs  into  viable  and  desired  outputs. 
This  should  lead  to  more  accurate  and  descriptive  conclusions  than  attempting  to  apply  more 
individually  defined  constructs  and  behavior  categorizations  to  team  interactions  and 
behaviors. 

Generally  speaking,  transition  processes  can  be  seen  as  those  behaviors  and  actions 
teams  enact  before  actually  performing  their  assigned  tasks.  Once  brought  together  to 
perform  a  task,  teams  and  individual  team  members  must  spend  some  time  to  gain  an 
understanding  of  their  situation,  the  tasks,  and  the  resources  they  have  available  to  them, 
including  team  member  expertise.  Each  of  these  behaviors  are  directed  at  employing  current 
inputs  to  arrive  at  decisions  and  conclusions  affecting  future  actions.  This  differs  from  more 
retrospective  sensemaking  behaviors  (c.g.,  Weick,  1995)  in  that  individuals  are  less 
concerned  about  making  sense  of  events  relative  to  themselves  and  their  sense  of  identity 
than  they  are  in  deciding  how  to  use  inputs  to  achieve  specified  task  goals.  Based  on 
previous  research  findings,  transition  process  behaviors  have  been  further  conceptualized  as 
mission  analysis,  goal  specification,  and  strategy  formation  (Marks  et  al.,  in  preparation). 

Mission  analysis  requires  team  members  to  interpret  and  evaluate  the  team’s  assigned 
tasks  or  mission.  This  includes  teams  identifying  the  mission’s  main  tasks  as.well  as  the 
operative  environmental  conditions  and  team  resources  available  for  mission  execution. 
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During  goal  specification,  teams  jointly  identify  and  prioritize  their  goals  and  subgoals  for 
accomplishing  their  overall  task  or  mission.  Teams  engage  in  strategy  planning  to  formulate 
their  plans  and  courses  of  actions  needed  to  accomplish  their  mission  goals.  According  to 
Marks  et  al.,  this  planning  includes:  (1)  making  a  primary  or  generic  plan,  (2)  exploring  and 
anticipating  how  to  react  to  the  most  probable  contingencies  that  could  occur  during  the 
mission,  and  (3)  setting  the  foundation  for  making  additional  changes  to  all  previous  plans  in 
response  to  unexpected  changes  in  the  situation  during  the  mission  (i.e.,  reactive  strategic 
planning). 

Action  processes  include  those  team  member  behaviors  exhibited  while  the  team  is 
carrying  our  their  planned  actions  and  accomplishing  performance  tasks.  According  to 
Marks  et  al.,  these  processes  include  monitoring  progress  toward  goals,  systems  monitoring 
(i.e.,  monitoring  internal  technology  systems  and  the  external  environmental  for  changes  and 
task  cues),  team  monitoring  and  backup  behavior,  coordination  activities,  and 
communications  between  team  members.  These  are  very  similar  to  the  teamwork  KSAs 
described  by  earlier  researchers  (cf.,  Cannon-Bowers  et  al.,  1995). 

Finally,  Interpersonal  processes,  as  described  by  earlier  researchers  (see  McIntyre  & 
Salas,  1995;  Pelled,  1996;  Prince  et  al.,  1997),  include  those  affect  driven  behaviors 
associated  with  reducing  interpersonal  conflict  and  keeping  team  members  motivated  (Marks 
et  al.,  in  preparation).  Teams  that  demonstrate  an  increased  awareness  of  potential  areas  of 
interpersonal  conflict  and  stereotyping  and  understand  how  these  factors  can  impact  overall 
team  performance  are  more  effective  performers,  regardless  of  specific  assigned  tasks 
(Pelled,  1996). 
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Although  focused  on  the  type  of  dynamic  task  environments  found  in  military 
settings,  Marks  et  al.  emphasized  that  their  taxonomy  capitalizes  upon  previous  research 
findings  and  can  be  generalized  to  many  other  task  environments.  They  asserted  that  the 
restructuring  of  previously  supported  taxonomies  into  a  framework  that  better  represents  the 
underlying  processes  operating  in  a  team  environment  provides  a  stronger  foundation  for 
future  research  and  performance  assessments.  They  also  emphasized  that  their  framework 
comprises  processes  and  behaviors  that  can  be  assessed  at  both  the  individual  (i.e.,  within 
teams)  and  team  (i.e.,  between  teams)  levels  of  measurement.  This  enhances  researchers’ 
ability  to  conduct  multilevel  assessments  of  team  performance  and  potentially  generalize 
their  results  to  various  work  environments. 

Outcomes  are  the  products  of  team  activity  that  are  valued  by  either  the  organization 
or  team  members.  These  outcomes  characterize  the  effectiveness  of  the  team  and  generally 
focus  on  the  quality,  quantity,  and  timeliness  of  the  products  the  team  produces  (Heffner, 
1996).  With  the  exception  of  the  initial  formation  of  the  team,  the  outcomes  from  one 
performance  episode  will  also  influence  the  inputs  of  the  next  performance  episode.  In  a 
dynamic  environment,  such  as  air  combat  operations,  teams  are  normally  required  to  produce 
multiple,  simultaneous  outcomes. 

Team  members  are  also  often  required  to  simultaneously  perform  individual  tasks 
while  fulfilling  team  responsibilities.  For  example,  pilots  and  weapons  officers  flying 
combat  aircraft  are  directly  responsible  for  simultaneously  identifying  and  tracking  many 
different  types  of  aircraft  and  ground  vehicles,  while  ensuring  the  team  fulfills  mission 
objectives  established  by  senior  commanders.  In  combat  environments,  a  relatively  high 
quantity  of  decision  outcomes  must  also  be  high  in  quality  to  prevent  or  minimize  losses 
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among  friendly  forces.  As  well  publicized  in  the  public  media,  a  mistake  in  identifying 
friend  from  foe  can  have  disastrous  consequences  for  the  team,  a  battle,  and  sometimes  the 
overall  war  effort. 

Based  on  this  theoretical  foundation  and  previous  research  empirically  employing 
variations  of  the  I-P-0  framework,  I  hypothesized  that  each  type  of  team  process,  i.e. 
transition,  action,  and  interpersonal,  would  directly  impact  how  well  teams  perform  the  task 
they  are  presented.  Based  on  existing  research,  the  link  between  interpersonal  processes  and 
effective  team  performance  may  be  weakest  and  include  some  contradictory  findings  (see 
Driskell  &  Salas,  1992).  However,  there  is  also  substantial  evidence  indicating  that 
interpersonal  interactions  play  an  important  role  in  planning  and  carrying  out  task  strategies 
(see  Pelled,  1996;  Prince  et  al.,  1997).  Based  on  the  framework  and  taxonomy  proposed  by 
Marks  et  al.  (in  preparation),  I  hypothesized  that: 

Hypothesis  1 :  Higher  levels  of  transition  process  behaviors  will  be  associated  with 
higher  levels  of  multi-team  system  (MTS)  performance. 

Hypothesis  2:  Higher  levels  of  action  process  behaviors  will  be  associated  with 
higher  levels  of  MTS  performance. 

Hypothesis  3:  Higher  levels  of  interpersonal  process  behaviors  will  be  associated 
with  higher  levels  of  MTS  performance. 

To  test  these  hypotheses,  this  study  placed  subjects  in  one  of  two  dyads,  representing 
operational  combat  aircrews  comprising  pilot  and  weapons  specialist  positions.  These 
positions  and  two-person  teams  were  jointly  responsible  for  achieving  assigned  mission 
objectives  in  three  simulated  air-to-ground  combat  missions.2  In  this  way,  team  members 
were  interdependent  within  their  assigned  dyad  as  well  as  within  the  larger  two  dyad  team, 
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referred  to  as  a  multi-team  system  (MTS).  Based  on  Marks  et  al.’s  (in  preparation) 
discussion,  this  two  dyad  structure  is  the  minimum  unit  possible  in  a  MTS.  This  setting  not 
only  lent  itself  to  advancing  the  existing  literature  on  team  process  and  performance  in  a 
multi-team  environment,  but  it  also  presented  an  excellent  opportunity  to  examine  team 
performance  variables  and  behaviors  in  terms  of  Marks  et  al.’s  taxonomy. 

One  consideration  for  the  hypotheses  tested  in  this  study  was  the  implied  ordering  of 
and  relationships  between  the  three  types  of  team  processes  described  by  Marks  et  al.  (in 
preparation).  Given  my  previous  description  of  these  constructs,  it  seemed  reasonable  to 
conclude  that  transition  processes  should  occur,  or  at  least  be  sampled  and  measured,  before 
teams  attempt  to  carry  out  their  plans,  i.e.  action  processes,  or  resolve  and  manage 
interpersonal  conflict  during  the  execution  of  their  actions,  i.e.  interpersonal  processes. 

Based  on  the  general  design  of  the  experimental  task,  i.e.  planning  and  carrying  out  a 
simulated  military  combat  mission,  teams  would  normally  meet  and  jointly  plan  their  actions 
and  activities  to  achieve  their  assigned  objectives,  i.e.  their  mission  strategy,  before  carrying 
out  any  actions  in  the  simulation.  While  it  is  certainly  reasonable  to  assume  that  teams 
would  engage  in  interpersonal  processes  during  these  planning  sessions,  I  decided  to  focus  on 
how  the  effectiveness  of  the  teams’  transition  processes  impacted  not  only  the  teams’  desired 
outcomes  (i.e.  hypothesis  1)  but  also  their  action  and  interpersonal  processes.  Based  on 
previous  research  and  personal  military  planning  and  training  experience,  I  hypothesized 
that: 

Hypothesis  4:  Higher  levels  of  transition  process  behaviors  will  result  in  increased 

levels  of  effective  team  action  process  behaviors. 
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Hypothesis  5:  Higher  levels  of  transition  process  behaviors  will  result  in  increased 
levels  of  effective  team  interpersonal  process  behaviors. 

Team  Effectiveness  reflects  how  well  a  team  accomplishes  its  purpose  or  mission 
(Tannenbaum,  Salas,  &  Cannon-Bowers,  1996;  Wiener,  Kanki,  &  Helmreich,  1993). 
Although,  as  illustrated  by  the  many  variations  described  in  the  literature,  effectiveness  can 
be  quantified  differently  for  almost  every  type  of  team  or  context,  almost  all  of  the  variations 
are  concerned  with  evaluations  of  a  team’s  collective  performance.  Research  and 
organizational  assessments  have  routinely  focused  on  the  quantity  and  quality  of  the  services 
and  products  provided  by  teams.  Aggregating  the  outputs  of  individual  team  members  or 
relying  on  an  archival,  objective  scoring  algorithm  frequently  define  team  outcomes  (see 
Cobb,  1998;  Elliott  et  al.,  1996;  Tesluk,  Mathieu,  Zaccaro,  &  Marks,  1997).  For  example, 
the  most  common  performance  measure  in  many  military  related  simulations  is  a  final 
performance  score,  usually  based  on  the  number  of  friendly  forces  surviving  and  the  number 
of  enemy  destroyed  during  the  simulation  (e.g.,  Cobb,  1998;  Elliott  et  al.,  1996;  Elliott, 
Neville,  Dalrymple,  &  Tower,  1997).  In  some  cases,  reliance  on  such  scoring  mechanisms 
may  present  an  incomplete  or  inaccurate  picture  of  the  team’s  overall  performance. 

There  is  little  information  in  objective,  automated  composite  scores  to  discern  how 
teams  are  able  to  overcome  unexpected  changes  in  the  situation  or  how  effectively  they 
prioritized  and  completed  their  assigned  tasks.  Destroying  “enemy”  targets  and  saving 
“friendly”  forces,  the  most  prevalent  factors  in  such  scoring  algorithms,  are  important 
criteria,  but  they  are  not  always  the  most  productive  outcomes  when  collective  actions  are 
required  (see  Cobb,  1998).  Not  everyone  on  a  team,  or  every  team  within  a  larger  grouping 
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or  team,  can  seek  to  maximize  their  individual  performance  score  without  undermining  the 
team’s  ability  to  achieve  its  overall  goals. 

Often,  especially  in  military  contexts,  such  egocentric  behavior  is  extremely 
counterproductive  and  dangerous  to  overall  mission  objectives  and  even  the  welfare  of  other 
assets  or  teams  working  in  the  same  environment  (i.e.,  joint  operations,  joint  ventures, 
research  and  development).  When  individuals  or  teams  pursue  their  own  goals,  while 
ignoring  those  associated  with  the  larger  collective  interest,  resources  become  depleted, 
strategies  are  undermined,  and  uncertainty  increases  since  others  can  no  longer  predict  their 
behavior  and  actions.  Tannenbaum  et  al.  (1996)  included  a  team’s  ability  to  overcome 
uncertainty,  “to  remain  vital  and  alive  and  to  grow  and  regenerate  itself,”  in  their  view  of 
team  effectiveness  (p.505).  They  emphasized  that  a  team’s  ability  to  overcome  uncertainty 
enables  a  team  to  sustain  its  performance  and  fulfill  its  purpose  over  a  period  of  time  despite 
increasingly  complex  task  environments. 

Environmental  Complexity. 

The  stability  and  nature  of  the  task  environment  a  team  operates  within  are  important 
contextual  factors  impacting  performance  (see  Guzzo,  1995;  Hackman  &  Morris,  1975;  Salas 
et  al.,  1992).  Based  on  his  review  of  current  research,  Guzzo  asserted  that  context  is  the 
primary  driver  of  team  effectiveness  (see  Guzzo,  1995;  Guzzo  &  Dickson,  1996).  Tasks  vary 
in  the  type  and  amount  of  interdependence  and  contextual  information  required  to 
successfully  complete  them  (see  Argote  &  McGrath,  1993;  Straus  &  McGrath,  1994).  The 
greater  the  need  for  team  members  to  depend  on  and  directly  support  one  another  to 
accomplish  tasks,  the  more  complex  their  interdependence  and  environment  will  be 
(Saavedra,  Earley,  &  Van  Dyne,  1993). 


17 


Interdependency  can  also  be  looked  at  as  the  interconnections  among  group  member 
tasks  (see  Van  de  Ven  &  Ferry,  1980).  If  a  specific  team  member  has  all  the  resources, 
information,  authority,  and  means  needed  to  make  and  carry  out  a  decision,  she/he  has  little 
reason  to  interact  with  other  team  members.  This  person’s  task  is  greatly  simplified,  since 
time  and  cognitive  resources  do  not  have  to  be  devoted  to  ensuring  a  teammate  understands 
the  situation,  agrees  to  the  intended  response,  and  provides  the  resources  or  assistance  needed 
to  fulfill  the  task.  Research  has  consistently  emphasized  the  importance  of  carefully 
considering  the  integration  of  individual  contributions  and  collective  coordination  when 
examining  team  and  group  behaviors  (see  Zaccaro,  Blair,  Peterson,  &  Zazanis,  1995). 

Using  low-fidelity  computer  simulations  and  two-person  teams  of  undergraduate 
students,  Stout,  Salas,  and  Carson  (1994)  examined  how  team  processes  and  communications 
were  impacted  by  task  complexity.  They  found  when  team  members  are  required  to  act 
interdependently,  behaviors  such  as  providing  information  before  it  is  needed,  planning, 
asking  for  input,  and  intervening  to  help  teammates  become  increasingly  important  to 
effective  task  performance.  There  has  been  a  significant  amount  of  research  indicating  that 
task  interdependence  affects  many  different  areas  of  group  and  team  performance  (Saavedra 
et  al.,  1993;  Shea  &  Guzzo,  1987),  as  well  as  such  key  team  processes  as  cooperation 
between  group  members  and  interpersonal  communication  and  interactions  between  team 
members  (Kelley  &  McGrath,  1985). 

In  their  study  of  task  strategies  and  intergroup  conflict,  Saavedra  et  al.  (1993) 
proposed  a  three-dimensional  construct,  complex  interdependence ,  to  capture  the  interactions 
between  task  interdependence,  goals,  and  feedback.  These  attributes  of  complex 
interdependence  corresponded  to  the  extent:  (a)  group  members  must  rely  on  each  other  to 
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perform  tasks,  (b)  individual  or  group  performance  objectives  are  provided  (goals),  and  (c) 
performance  feedback  is  provided  for  individual  members  or  the  entire  group  (feedback) 
(Saavedra  et  al.,  1993).  Based  on  earlier  research  (see  Thompson,  1967),  they  proposed  that 
there  exists  a  hierarchy  of  group  member  interactions  that  reflects  increasing  levels  of 
dependence  and  coordination  needs.  At  the  lowest  level,  pooled  interdependence  requires  no 
direct  interaction  between  individuals  as  group  members  fulfill  similar  roles.  While  each 
individual  contributes  to  the  overall  performance  of  the  group,  group  members  complete 
assigned  tasks  largely  without  regard  for  the  work  of  others. 

The  next  level,  according  to  Saavedra  et  al.  (1993),  is  sequential  interdependence.  At 
this  level,  one  group  member’s  output  requires  the  input  of  another  in  a  sequential  fashion. 
Group  members  occupy  different  roles  and  perform  different  tasks  in  a  prescribed  order.  At 
the  next  higher  level,  reciprocal  interdependence,  one  person’s  output  also  becomes  another’s 
input,  but  the  order  is  flexible.  Group  performance  requires  coordination  of  differing  roles 
and  specialties  among  its  members.  Finally,  team  interdependence  forces  group  members  to 
jointly  problem  solve  and  collaborate  to  complete  tasks.  This  requires  interaction  to 
determine  the  particular  course  of  inputs  and  outputs  among  members  and  usually  involve  an 
exchange  of  resources,  information,  or  ideas  (Saavedra  et  al.,  1993;  Tesluk  et  al.,  1997). 

While  there  has  been  considerable  research  that  has  focused  on  the  level  of 
interdependence  required  by  or  demonstrated  by  team  activities,  there  has  actually  been  very 
little  research  that  has  sought  to  directly  vary  the  complexity  of  the  interdependent  tasks 
themselves.  According  to  Kluger  and  DeNisi  (1996),  complexity  reflects  the  number  of 
actions,  dependencies  among  actions,  and  the  temporal  dependencies  needed  for  successful 
task  performance.  Complexity  is  best  considered  as  a  task  characteristic  independent  of  the 
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attributes  of  the  participants  performing  it  (Wood,  1986).  In  this  perspective,  complexity 
arises  directly  from  the  nature  of  the  task  and  the  task  environment,  not  from  the  attributes  or 
interactions  of  the  individuals  accomplishing  the  task.  It  is  only  when  the  actions  of  an 
individual,  or  team  in  the  context  of  this  proposal,  alters  the  nature  of  the  task  or  the  physical 
environment  in  such  a  way  as  to  change  the  acts  required  to  accomplish  the  task  that 
individuals  contribute  to  the  complexity  of  the  situation.  In  his  classic  examination  of  the 
construct,  Wood  (1986)  describes  three  types  of  complexity:  component,  coordinate,  and 
dynamic. 

Component  complexity  is  concerned  with  the  number  of  distinct,  nonredundant  acts 
that  must  be  executed  to  perform  the  task.  Component  complexity  is  also  concerned  with 
how  many  information  cues  must  be  processed  when  performing  the  activity  (Wood,  1986). 
As  can  be  seen  in  research  examining  situational  awareness  (see  Endsley,  1995;  Goodwin, 
1997),  work  environments  differ  greatly  in  the  specific  features  a  team  must  attend  to  in 
order  to  succeed.  The  ability  to  detect  and  distinguish  the  presence  or  absence  of  various 
environmental  cues  is  an  important  consideration  in  decision  making  behaviors.  The  greater 
the  combination  of  object  features  to  which  a  subject,  an  individual  or  team,  must  attend,  the 
more  difficult  the  decision  task  (Kaempf  et  al.,  1996;  Triesman,  1991). 

Kaempf  et  al.  (1996)  found  that  95%  of  their  subjects  used  recognition  strategies 
when  choosing  decision  options  in  complex,  time  constrained  situations.  Their  subjects 
relied  on  feature-matching  and  story  generation  strategies  to  build  and  maintain  their 
situational  awareness.  As  many  other  researchers  had  discovered  (see  Meller,  Schwartz,  & 
Cooke,  1998;  Oaksford  &  Chater,  1994;  Tversky  &  Kahneman,  1982),  Kaempf  et  al. 
described  how  their  subjects  relied  on  heuristics  and  satisficing  (i.e.,  selecting  the  first 
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adequate  solution)  techniques  to  reduce  their  cognitive  workload  and  deal  effectively  with 
the  complexity  of  their  decision  tasks. 

From  a  military  standpoint,  one  of  the  goals  of  training  is  to  ensure  that  individuals 
are  capable  of  distinguishing  friendly  from  enemy  forces.  In  the  past,  military  members  were 
trained  to  recognize  key  features  that  distinguished  one  weapon  design  from  another.  This 
feature-based  strategy  was  effective  as  long  as  potentially  competing  nations  employed 
differing  weapons  designs.  In  his  study  of  problem-solving  behavior  among  commercial 
pilots,  Casner  (1994)  found  that  pilots  relied  heavily  on  feature-matching  strategies  when 
reacting  to  air  traffic  controller  inputs.  Similar  results  were  found  by  Kaempf  et  al.  (1996) 
in  their  observations  of  decision  processes  on  US  Navy  AEGIS  cruisers.  Due  to  the 
increased  presence  of  similar,  if  not  identical,  military  hardware  throughout  the  world, 
matching  features  to  designs  and  using  these  matches  to  determine  their  nationality  has 
become  an  increasingly  difficult  task  in  the  modem  battlefield.  In  essence,  increased 
similarity,  if  not  complete  redundancy  due  to  reverse  engineering  or  international  weapons 
trade,  between  the  weapons  used  by  competing  nations  has  undermined  previous  training 
based  on  identifying  and  matching  specific  design  features. 

Wood  (1986)  also  identified  two  other  types  of  complexity.  Coordinative  complexity 
focuses  on  the  timing  or  sequencing,  frequency,  intensity,  and  location  required  to  perform 
required  acts.  The  more  complex  the  relationships  between  these  attributes,  the  greater  the 
knowledge  and  skill  an  individual  must  have  to  accomplish  the  task.  Dynamic  complexity  is 
created  by  changes  in  the  environment  that  impact  the  relationship  between  task  inputs  and 
task  products.  Since  performance  in  such  complex  environments  requires  knowledge  about 
changes  over  time,  it  might  be  easier  to  equate  this  type  of  complexity  with  the 
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predictiveness  of  the  environment.  Lower  levels  of  dynamic  complexity  are  characterized  by 
stable,  predictable  environments.  Higher  levels  of  complexity  exist  when  unexpected 
changes  alter  the  performance  environment  and  the  relationships  between  task  inputs  and 
products. 

The  ability  to  identify  and  remain  cognizant  of  changes  in  the  environment  and 
relevant  task  demands,  i.e.  situational  awareness,  is  a  critical  performance  component  of 
highly  dynamic  and  complex  tasks  (Goodwin,  1997;  Wellens,  1993).  Situational  awareness 
includes  perceiving  current  environmental  elements  in  both  time  and  space,  comprehending 
the  meaning  and  significance  of  these  elements,  and  predicting  probable  actions  of  these 
elements  in  the  immediate  future  (Endsley,  1995).  Research  and  experience  have  repeatedly 
shown  that  situational  awareness  is  fundamental  to  successful  team  performance  in  military, 
organizational,  and  industrial  settings.  Kozlowski  &  Gully  (1996)  estimated  that  80  percent 
of  aircraft  accidents,  as  well  as  tragic  events  like  the  destruction  of  a  civilian  airliner  by  the 
USS  Vincennes  in  the  Persian  Gulf,  are  attributable  to  errors  made  by  highly  trained, 
experienced  specialists  operating  in  a  team  context.  Official  investigators  and  academic 
researchers  have  tracked  many  of  these  errors  to  a  breakdown  in  a  team’s  ability  to  recognize 
and  adapt  to  unexpected  changes  in  the  environment. 

Although  it  has  been  frequently  studied  by  military  researchers  in  terms  of  aircraft 
cockpit  designs  and  head-up  displays  (e.g.,  Goodwin,  1997;  Stinnet,  1989),  situational 
awareness  has  gained  in  importance  when  studying  team  process.  For  example,  enhancing 
individual  and  collective  situational  awareness  has  become  increasingly  significant  in  pilot 
and  aircrew  training  programs  (Shrestha,  Prince,  Baker,  &  Salas,  1995).  Yet,  as  Mathieu, 
Goodwin,  Heffner,  and  Bom  (1996)  pointed  out,  there  has  been  relatively  limited  research 
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focusing  on  the  role  of  contextual  influences  on  team  processes.  While  some  researchers 
have  emphasized  the  importance  of  accounting  for  the  impact  of  organizational  context  or 
environment  on  team  effectiveness  (see  Gladstein,  1984;  Salas  et  al.,  1992),  Mathieu  et  al. 
noted  that  even  among  the  few  studies  empirically  examining  this  issue,  teams  are  seen  as 
largely  passive  respondents  to  situational  demands. 

Following  Ancona  and  her  colleagues’  research  indicating  that  teams  are  affected  by 
and,  in  turn,  affect  their  environment  (Ancona  &  Caldwell,  1988, 1992),  Mathieu  et  al. 

(1996)  examined  the  hypothesis  that  team  processes  and  environmental  demands  co-evolve 
over  time.  They  determined  that  as  teams  choose  task  strategies  and  execute  decisions  in 
response  to  diverse  situational  demands,  their  actions  alter  the  resources  available  for  future 
actions  and,  to  some  degree,  the  environment  surrounding  them.  For  example,  an  Airborne 
Warning  and  Control  Systems  (AW ACS)  crew  vectoring  friendly  resources  to  respond  to  an 
enemy  threat  not  only  changes  the  friendly  resources  available  for  future  responses,  but  the 
success  or  failure  of  their  decision  also  impacts  possible  reactions  by  the  adversary.  Thus,  an 
action-reaction  cycle  begins  that  directly  impacts  the  potential  complexity  of  future  situations 
and  the  very  nature  of  any  further  threat  options. 

From  this  discussion,  one  would  expect  teams  to  have  more  difficulty  effectively 
performing  in  complex  situations  than  in  more  routine,  normal  conditions.  By  their  nature, 
complex  situations  place  more  task  demands  upon  individuals  and  teams  and  require  higher 
levels  of  problem  solving  than  more  routine  conditions  (cf.,  Funke,  1991;  Kaempf  et  al., 

1996;  Orasanu,  1993).  Based  on  the  existing  literature  (cf.,  Cobb,  1998;  Elliott  et  al.,  1996; 
Goodwin  1997;  Guzzo  &  Dickson,  1996),  I  expect  higher  levels  of  complexity  to  negatively 
impact  MTS  performance. 
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Hypothesis  6:  Increased  levels  of  environmental  complexity  will  have  a  direct 
negative  impact  on  teams’  performance  in  multi-team  environments. 

The  situation  can  become  even  more  convoluted,  when  one  considers  the  fact  that  the 
AW  ACS  crew  mentioned  previously  is  only  one  of  many  teams  operating  in  the  environment 
with  the  capability  to  direct  and  redirect  combat  resources.  The  combat  forces,  themselves, 
are  also  comprised  of  teams  highly  skilled  in  the  application  of  force  in  each  of  the  combat 
environments,  i.e.  land,  air,  and  sea.  All  of  these  combat  and  control  teams  must  be  carefully 
integrated  into  a  comprehensive  plan  that  orchestrates  their  actions  within  a  larger  context  to 
achieve  mission  goals  and  command  objectives.  While  the  actions  of  individual  aircrews  and 
combat  units  are  important,  it’s  the  performance  of  the  larger  “team  of  teams”  that  ultimately 
determines  the  success  or  failure  of  the  overall  mission.  During  training,  individuals  not  only 
learn  about  their  individual  responsibilities  on  their  specific  aircrew  or  combat  unit,  but  they 
are  also  trained  to  understand  how  their  team  fits  into  larger  combined  forces  -  in  essence, 
mission  teams.  These  training  experiences  are  thought  to  provide  the  task  and  teamwork 
KSAs  needed  to  function  and  succeed  in  increasingly  complex  environments. 

Team  Training. 

Kozlowski  and  Gully  (1996)  asserted  that  most  training  programs  are  based  on 
making  the  right  decision  to  achieve  some  specified  outcome.  They  challenged  this  notion 
by  emphasizing  that  rational  analysis  does  not  always  ‘...  yield  optimal  solutions  when  the 
problem  is  ill-defined,  information  is  ambiguous,  and  the  situation  is  dynamic”  (p.4).  Their 
focus  was  on  assessing  the  appropriateness  of  the  team’s  overall  decision  process,  i.e.  making 
the  decision  in  the  right  way.  Although  they  concentrated  on  the  adaptive  expertise  of  teams 
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from  a  learning  perspective,  Kozlowski  and  Gully  made  a  very  valid  point.  Real  world 
situations  and  problems  rarely  fit  rational  models  of  outcome-based  decision  models. 

While  a  strength  of  their  research  was  in  linking  active  learning  processes  with 
adaptive  expertise,  Kozlowski  and  Gully  (1996),  as  well  as  other  researchers,  still  seemed  to 
ignore  how  team  decisions  can  radically  change  their  environmental  context.  To  put  it 
another  way,  environments  are  not  only  dynamic  and  fluid,  but  they  are  also  continually 
evolving.  Training  teams  for  such  dynamic  environments  requires  team  members  to  master 
situationally  relevant  knowledge,  skills,  and  attitudes  (KSAs),  as  well  as  developing  more 
general  skills  that  can  be  transferred  to  other  task  environments  and  team  settings  (Cannon- 
Bowers  et  al.,  1995). 

Crew  resource  management  (CRM)  training  is  specifically  designed  to  enhance  the 
effectiveness  of  aviation  crews’  team  process  by  improving  communication,  coordination, 
and  situational  awareness  (Cavanagh  &  Williams,  1987;  Guzzo  &  Dickson,  1996).  This 
type  of  training  is  not  intended  to  strengthen  any  particular  team.  Rather,  CRM  seeks  to 
make  individuals  more  effective  in  team  settings  by  emphasizing  team  process  strategies  and 
behaviors.  Recognizing  the  broad  scope  and  variety  of  training  activities  involved  in  civilian 
and  military  CRM  programs,  Bowers,  Morgan,  Salas,  and  Prince  (1993)  described  how 
aircrew  coordination  training  (ACT)  focuses  on  training  interventions  specifically  designed 
to  improve  coordination  behaviors  required  for  successful  flight  operations.  This  type  of 
CRM  training  has  been  especially  embraced  by  the  U.S.  Navy  to  improve  the  quality  and 
effectiveness  of  undergraduate  pilot  and  aircrew  training. 

In  essence,  CRM  training  and  ACT  focus  on  developing  behaviors  and  KSAs  directly 
relevant  to  aircrew  coordination  and  communication  under  many  different  task  conditions 
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(see  Prince  &  Salas,  1993).  These  programs  are  concerned  with  team  members  mastering 
what  Cannon-Bowers  et  al.  (1995)  called  generic  team  competencies.  Contrary  to  skills  and 
knowledge  required  to  effectively  function  as  a  member  of  one  specific  team,  these  generic 
competencies  should  be  applicable  to  virtually  any  team  setting  or  composition  (Cannon- 
Bowers  et  al.,  1995).  Given  the  transitory  nature  of  aircrew  membership  in  civilian  and 
military  settings,  identifying  and  emphasizing  transferable  team  process  skills  is  an  important 
aspect  of  CRM  and  other  forms  of  team  training. 

Due  to  manning  limitations  and  scheduling  demands,  it  has  been  common  practice  for 
most  military  teams,  especially  combat  teams  and  aircrews,  to  frequently  alter  their 
membership  over  time.  As  in  commercial  industries  and  airlines,  some  teams  may  only  be 
brought  together  for  a  single  task  or  mission.  Once  that  is  accomplished,  the  individuals 
return  to  their  original  units  or  departments.  In  some  military  situations,  especially  during 
long  deployments  and  combat  engagements,  team  membership  may  change  many  times  in  a 
relatively  short  time  during  the  duration  of  the  mission  deployment. 

Researchers  (e.g.,  Leedom  &  Simon,  1995)  have  shown  that  frequently  altering  air 
crew  membership  may  actually  minimize  the  number  of  performance  errors  committed  by 
crews  due  to  complacency  and  overconfidence.  Leedom  and  Simon  (1995)  found  that 
established  military  practices  emphasizing  flexible  crew  pairings,  rather  than  fixed,  long-term 
crew  pairings  (i.e.,  battle  rostering),  actually  seem  to  enhance  team  performance.  They  also 
determined  that  behavior-based  coordination  training  resulted  in  significantly  more  effective 
team  performance  and  fewer  mission  errors  than  reliance  on  team  member  familiarity  or 
more  attitudinal  based  training  interventions. 
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Regardless  of  the  timing  of  the  change  or  the  duration  of  membership,  military 
commanders  and  organizational  managers  fully  expect  individuals  to  clearly  understand 
his/her  role  and  position  within  the  team.  New  team  members  are  expected  to  be  completely 
capable  of  not  only  carrying  out  their  assigned  tasks,  but  also  of  smoothly  interacting  with 
other  members  of  the  team  on  an  interpersonal  basis.  Such  a  perspective  relies  heavily  on 
training  programs  to  teach  individuals  how  to  integrate  taskwork  and  teamwork  skills  (Salas 
etal.,  1992). 

According  to  Cannon-Bowers  et  al.  (1995),  taskwork  skills  include  those  KSAs 
needed  to  apply  these  task  competencies  to  execute  assigned  tasks  and  responsibilities. 

These  include  cue-strategy  associations  and  understanding  team  member  roles  and 
responsibilities.  Teamwork  KSAs  enable  team  members  to  actually  carry  out  required 
functions  and  tasks.  These  competencies  include  such  behaviors  as  situational  awareness, 
performance  monitoring  and  feedback,  coordination,  and  communication  (Cannon-Bowers  et 
al,  1995). 

Similar  to  the  taxonomy  suggested  by  Cannon-Bowers  et  al.  (1995)  that  emphasized 
task  and  team  competencies,  Salas  et  al.  (1992)  asserted  that  embedding  important  task 
information  within  relevant  teamwork  skills  may  be  an  effective  training  approach.  Marks  et 
al.  (1997)  presented  subjects  with  essential  task  elements  accompanied  by  structures  of  team 
interaction.  Teams  were  provided  information  about  the  relationships  between  components, 
including  role  structures  associated  with  individual  functions,  and  the  relationships  between 
role  structures.  They  found  a  significant  relationship  between  teamwork  training  and  team 
effectiveness. 
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Based  on  their  extensive  review  of  CRM  research,  critical  incidents  interviews  with 
over  200  military  cockpit  crew  members,  and  analysis  of  225  aviation  mishaps,  Prince  and 
her  colleagues  concluded  that  seven  skills  appeared  to  be  particularly  critical  to  effective 
team  process  and  performance  (Prince  et  al.,  1997;  Prince  and  Salas,  1993).  They  identified 
the  following  skill  dimensions  as  especially  critical  to  aviation  teams:  (1)  communication, 
(2)  decision  making,  (3)  leadership,  (4)  situation  awareness,  (5)  mission  analysis,  (6) 
assertiveness,  and  (7)  adaptability/flexibility.  Analysis  of  the  responses  of  134  aviators’ 
responses  to  the  proposed  dimensions  also  indicated  these  categorizations  captured  the  vital 
aspects  of  aircrew  coordination  training  (ACT).  Unlike  Fleishman  and  Zaccaro’s  (1992) 
taxonomy,  Prince  et  al.’s  categorization  focuses  on  the  interaction  patterns  of  the  team 
members  rather  than  coordinated  task  performance  outcomes. 

Much  of  the  USAF’s  research  on  combat  air  crews  and  airborne  command-and- 
control  teams  has  also  emphasized  the  importance  of  these  particular  skill  dimensions  in 
developing  effective  team  processes  and  enhancing  team  performance  (see  Cobb,  1998; 
Elliott  et  ah,  1996;  Elliott,  Neville,  Dalrymple,  and  Tower,  1997).  While  measuring  the 
collective  impact  of  these  dimensions  has  presented  problems  for  analysis,  Prince  et  ah 
(1997)  have  found  that  team  process  measures  can  sometimes  be  effectively  reduced  to  a 
single  omnibus  process  score.  Heffner  (1996)  used  a  similar  aggregated  construct  to  analyze 
team  process  in  her  examination  of  the  impact  of  training  on  team  mental  models. 

There  has  been  an  ongoing  debate  in  the  literature  as  to  whether  the  decision  to  focus 
on  a  single  omnibus  measure  (see  Prince  et.  ah,  1997;  Heffner,  1996)  or  to  separate  behaviors 
into  individual  evaluations  is  more  effective  in  testing  hypotheses  (see  Cannon-Bowers  et. 
ah,  1992).  It  is  not  really  clear  in  the  debate  if  the  issue  is  a  matter  of  measurement  or 
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construct  assessment.  The  basic  question,  in  my  opinion,  is  whether  there  is  a  difference  in 
assessing  team  behaviors  in  terms  of  their  individual  effect  or  as  integrated  components  of  a 
greater  whole,  such  as  team  process.  There  seems  to  be  much  support  for  both  camps. 
Further  research  is  required  to  examine  the  theoretical  implications  of  both  positions. 

In  summarizing  the  existing  research  on  team  training,  Salas  et  al.  (1992)  pointed  out 
that  informing  team  members  of  the  nature  and  requirements  of  other  team  members’ 
subtasks  is  an  effective  way  to  emphasize  the  need  for  communication  and  coordination. 

This  cross  training  encourages  team  members  to  identify  their  interdependencies,  recognize 
when  and  to  where  information  must  be  transferred,  and  understand  the  consequences  of 
failing  to  effectively  coordinate  their  respective  task  responsibilities  and  decisions. 
According  to  researchers  examining  such  cross  training  strategies  (e.g.,  Cannon-Bowers  et 
al.,  1992;  Stout,  Cannon-Bowers,  Morgan,  &  Salas,  1990;  Stout,  Salas,  &  Carson,  1994; 
Volpe,  Cannon-Bowers,  &  Salas,  1996),  the  ability  of  one  team  member  to  anticipate  the 
needs  of  another  significantly  enhances  overall  team  performance. 

Salas  et  al.  (1992)  asserted  that  effective  teams  operating  under  high  stress  conditions 
employ  implicit  coordinating  mechanisms.  These  mechanisms  are  implicitly  understood  by 
the  team  members  and  require  only  minimum  interaction  within  the  team  to  coordinate 
specific  actions.  These  behaviors  and  shared  perceptions  allow  team  members  to  anticipate 
others’  needs  while  reducing  the  need  for  overt  communication.  When  the  unexpected 
occurs,  or  there  are  no  standard  procedures  to  follow,  or  a  team  member  does  not  have  the 
expertise  needed  for  the  task,  interaction  with  other  team  members  becomes  essential  for 
making  effective  decisions.  Effective  training  enables  team  members  to  both  develop  these 
implicit  strategies  and  understand  how  to  react  when  they  fail. 
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While  there  has  been  extensive  research  into  the  basic  principles  of  team  performance 
and  training  designs  (cf.,  Cannon-Bowers,  Tannenbaum,  Salas,  &  Volpe,  1995;  Morgan, 
Glickman,  Woodward,  Blaiwes,  &  Salas,  1986;  Prince,  Brannick,  Prince,  &  Salas,  1997), 
Salas,  Bowers  and  Cannon-Bowers  (1995)  pointed  out  that  there  has  been  almost  no  research 
that  has  focused  on  directly  testing  team  training  interventions.  Research  on  training 
interventions  has  been  concerned  with  comparing  the  effectiveness  of  various  methods  of 
conducting  team  training,  e.g.  video-based,  lecture,  information-based,  behavior  modeling, 
attitudinal,  practice-based,  individual  vs.  team  training,  whole  vs.  part  training,  and 
simulations  (e.g.,  Andrews,  Waag,  &  Bell;  Bowers  et  al.,  1993;  Dwyer,  Oser,  Salas,  & 
Fowlkes,  1999;  Salas  et  al.,  1992;  Weiner,  1993),  and  developing  adequate  methods  of 
training  assessment  (e.g.,  Fowlkes  et.  al.,  1994;  Gregorich  &  Wilhelm,  1993;  McIntyre  & 
Salas,  1995;  O’Neil,  Baker,  &  Kazlauskas,  1992).  Salas  et  al.  (1995)  emphasized  that  these 
research  efforts  examining  team  training,  especially  in  military  settings,  has  mainly  focused 
on  developing  models  and  taxonomies  for  developing  team  training  programs.  They  pointed 
out  while  this  research  has  provided  a  reasonable  database  for  making  preliminary 
recommendations  to  training  designers,  there  is  still  much  research  to  be  done.  Included  in 
their  recommendations  is  a  call  for  empirical  tests  of  various  training  strategies  and 
determining  how  they  are  linked  to  team  performance. 

Cannon-Bowers  et  al.  (1995)  asserted  that  training  must  consider  the  competencies 
required  for  effective  teamwork  and  the  context  in  which  the  team  performs.  Different 
instructional  strategies  depend  on  the  team’s  context,  the  nature  of  the  task,  and  the 
composition  and  characteristics  of  the  team.  In  operational  military  settings,  aircrew  training 
first  builds  a  foundation  of  knowledge  needed  to  understand  how  a  particular  aircraft  or 
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weapon  system  operates.  As  suggested  by  many  researchers  (cf.,  Cannon-Bowers  et  al., 
1995;  Marks  et  al.,  in  preparation;  Stout  et  al.,  1997)  and  required  by  established  military 
training  directives,  military  trainers  rely  on  detailed  task  analyses  of  what  KSAs  are  required 
for  military  members,  including  aircrews,  to  effectively  carry  out  their  assigned 
responsibilities.  As  observed  by  Stout  et  al.  (1997)  in  their  study  of  Navy  aviation  trainees, 
students  progress  through  increasingly  difficult  tasks  and  examinations  as  they  complete  their 
training  program.  In  operational  military  aircrew  training,  important  considerations  include 
not  only  the  individual  and  team  competencies  required  to  fly  the  aircraft,  but  also  the  KSAs 
required  to  plan  military  combat  operations  to  fulfill  mission  objectives.  It  is  accomplishing 
these  specified  goals  that  ultimately  determine  the  success  or  failure  of  any  military 
operation. 

Marks  et  al.’s  (in  preparation)  taxonomy  emphasized  the  importance  of  planning 
skills  (i.e.,  mission  analysis,  goal  specification,  and  strategy  formulation  and  planning)  as 
transition  processes.  Team  communication,  coordination,  and  other  monitoring  and  back-up 
behaviors  are  central  to  action  processes  in  their  framework.  Based  on  previous  research  (cf., 
Cannon-Bowers  et  al.,  1992;  Cannon-Bowers  et  al.,  1995;  Casner,  1994;  Cavanagh,  & 
Williams,  1987;  Stout  et  al.,  1997),  training  focused  on  improving  team  member  competence 
in  these  processes,  and  thus  enhance  overall  performance,  must  address  planning  or  problem¬ 
solving  KSAs  in  addition  to  more  traditional  KSAs  characterizing  team  action  process 
behaviors. 

Based  on  Cannon-Bowers  et  al.’s  (1995)  conclusions  and  Marks  et  al.’s  description, 
training  on  how  to  plan  and  formulate  team-based  strategies  to  achieve  specified  objectives 
should  enhance  a  team’s  transition  processes.  As  seen  in  earlier  research  (see  Leedom  & 
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Simon,  1995),  training  focused  on  improving  team  coordination  behaviors  should  improve 
teams’  action  processes.  As  hypothesized  earlier,  this  improvement  of  process  behaviors 
should  ultimately  result  in  higher  levels  of  performance. 

Hypothesis  7:  Teams  receiving  strategy  training  will  exhibit  more  effective  transition 
process  behaviors  than  other  teams. 

Hypothesis  8:  Teams  receiving  coordination  training  will  exhibit  more  effective 
action  process  behaviors  than  other  teams. 

Due  to  the  generally  higher  levels  of  ambiguity  associated  with  complex  situations, 
training  interventions  should  both  impact  and  be  impacted  by  higher  levels  of  environmental 
complexity.  On  the  one  hand,  training  should  enhance  teams’  ability  to  cope  with  increasing 
levels  of  complexity  in  the  task  environment.  On  the  other  hand,  increased  levels  of 
complexity  should  threaten  to  overwhelm  the  sometimes  limited  performance  options 
developed  by  team  members  during  training.  Based  on  operational  military  experience  and 
the  research  previously  reviewed  in  this  paper,  this  is  especially  true  in  conditions  where 
teams  must  master  and  perform  their  tasks  within  a  short  time  period.  Therefore,  I 
hypothesize  that: 

Hypothesis  9:  There  will  be  less  difference  in  the  performance  of  teams  receiving 
strategy  training  versus  those  that  do  not  (i.e.,  control)  at  lower  levels  of  complexity 
than  at  higher  levels. 

Hypothesis  10:  There  will  be  less  difference  in  the  performance  of  teams  receiving 
coordination  training  and  those  that  do  not  (i.e.,  control)  at  lower  levels  of  complexity 
than  at  higher  levels. 
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Hypothesis  1 1 :  Teams  receiving  strategy  training  will  perform  better  in  more 
complex  situations  than  teams  in  the  control  condition. 

Hypothesis  12:  Teams  receiving  coordination  training  will  perform  better  in  more 
complex  situations  than  teams  in  the  control  condition. 

Figure  2  presents  an  illustrative  summary  of  the  hypotheses  tested  in  this  study. 
Figure  2:  Summary  of  Hypotheses 
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Chapter  3 

METHODS 

Participants. 

Due  to  the  large  number  of  subjects  required  to  complete  enough  four-person  teams 
(i.e.,  two  dyads  for  each  MTS  experimental  session)  to  test  the  hypotheses  described  in  this 
study,  participants  were  recruited  from  two  undergraduate  subject  populations: 
undergraduate  psychology  students  and  Reserve  Officer  Training  Corps  (ROTC)  cadets  and 
midshipmen.4  Ideally,  this  study  would  have  only  used  volunteer  participants  from  the  US 
Air  Force  (USAF),  Navy,  and  Army  ROTC  programs.  Due  to  their  unique  military  and 
academic  training,  ROTC  students  potentially  have  greater  experience  in  team-based  tasks, 
command  relationships,  leadership,  strategy  formation,  and  task  allocation  than  average 
undergraduate  students.  However,  even  with  the  direct  support  of  the  senior  officer 
command  staffs,  there  were  not  enough  volunteers  from  this  sample  base  to  fully  test  the 
hypotheses  in  this  study.  It  was  necessary  to  complete  the  design  with  additional  teams 
comprised  of  undergraduate  psychology  students. 

As  can  be  seen  in  Figure  2  (p.33),  the  proposed  model  theorizes  about  how 
manipulating  three  independent  factors  (i.e.,  two  training  conditions  and  environmental 
complexity)  will  directly  and  indirectly  affect  four  dependent  variables  (i.e.,  MTS 
performance,  interpersonal  process  behaviors,  action  process  behaviors,  and  transition 
process  behaviors).  The  proposed  model  employed  a  2  x  3  x  3  mixed  factorial  and  repeated 
measures  design  crossing  environmental  complexity  (high  vs.  low)  with  training  conditions 
(strategy,  coordination,  control)  over  three  experimental  sessions.  Using  both  ROTC  and 
undergraduate  psychology  students  provided  36  MTSs,  i.e.  6  MTSs  in  each  experimental 
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design  cell  crossing  complexity  and  training  conditions,  to  test  the  hypotheses  summarized  in 
the  proposed  model. 

Overall  Participant  Demographics.  Performance  and  process  measures  were  gathered 
for  36  four-person  teams  (N  =  144).  Reflecting  the  emphasis  placed  on  their  recruitment  for 
this  study,  the  majority  of  participants  were  undergraduate  ROTC  students  (92;  63.9%). 
Students  current  enrolled  in  undergraduate  psychology  courses  comprised  48  (33.3%)  of  the 
remaining  participants.  Table  1  provides  a  summation  of  the  subject  population  assigned  to 
each  experimental  condition.  Work  continued  on  the  computer  simulations  and  presentations 
supporting  additional  experimental  conditions  as  subjects  were  being  observed  in 
coordination  training  and  low  complexity  conditions,  which  were  the  first  to  be  developed. 
Teams  were  randomly  assigned  to  experimental  conditions  as  they  became  available. 
Performance  scores  from  single-limb  and  dual-limb  visual  tracking  tests  developed  by  USAF 
researchers  at  Brooks  Laboratory,  San  Antonio  TX,  were  used  to  assign  subjects  to  each  dyad 
(i.e.,  lead  or  wing  aircraft)  and  their  aircrew  position  (i.e.,  pilot  or  weapons  specialist). 

Reflecting  the  male-dominated  composition  of  the  ROTC  detachments,  68.8%  of  the 
participants  were  male,  while  27.1%  were  female.  Six  participants  in  the  study  failed  to 
designate  their  sex  during  preliminary  biodata  collection.  The  participants  ranged  in  age 
from  18  to  31  (mean  =  20.04  years).  The  majority  of  participants  identified  themselves  as 
Caucasian  (79.2%)  while  the  remainder  were  almost  equally  divided  into  several  minority 
groups  (Asian  American:  6.9%;  African-American:  4.9%;  Hispanic:  4.2%)  (see  Table  2). 

There  was  also  one  four-person  team  participating  in  this  study  composed  of 
undergraduate  student  members  of  a  local  gaming  association.  These  individuals  were 
interested  in  participating  due  to  their  interest  in  gaming  simulations,  such  as  the  one  used  in 
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this  experiment.  Although  experienced  in  computer  games  of  many  different  varieties, 
discussions  with  the  team  members  indicated  this  was  their  first  experience  with  air  combat 
simulations.  Due  to  their  low  number  and  greater  similarities  with  the  ROTC  sample 
population,  especially  given  their  greater  familiarity  with  each  other,  it  was  decided  to 
include  this  team  in  the  ROTC  sample  for  all  analyses  in  this  study.  To  further  minimize 
their  potential  impact  on  the  study’s  design,  they  were  assigned  to  the  control  condition. 

ROTC  Participants.  After  discussing  the  potential  training  benefits  and  the  general 
nature  of  the  study  with  each  of  the  ROTC  detachment  commanders,  I  was  permitted  to 
recruit  volunteers  from  each  of  the  three  ROTC  units  -  U.S.  Air  Force,  U.S.  Navy,  and  U.S. 
Army.  Additionally,  I  demonstrated  the  simulation  to  staff  representatives  from  each  service 
detachment.  This  demonstration  provided  each  of  the  detachment  commanders  with  a  better 
understanding  of  the  nature  of  the  experiment  and  the  potential  benefits  it  held  for  enhancing 
their  military  training.  Although  the  simulation  used  in  this  study  was  most  relevant  to  Air 
Force  training  and  education  objectives,  each  of  the  commanders  decided  to  support  the 
study  and  recruit  volunteers  from  their  cadet  and  midshipmen  populations. 

The  commander  of  each  detachment  appointed  a  representative  to  act  as  recruiter  and 
schedule  coordinator  for  the  study.  This  individual  was  responsible  for  publicizing  the  study 
among  the  ROTC  students,  signing  up  volunteers,  and  working  with  the  experimenters  to 
schedule  experimental  sessions.  While  the  Air  Force  and  Army  detachment  representatives 
were  students  who  were  enrolled  in  the  program  and  interested  in  participating  in  the 
experiment,  the  Navy  volunteer  was  their  staff  officer  responsible  for  freshmen  courses  and 
military  training.  As  one  might  have  expected,  since  the  computer  simulation  used  in  this 
study  illustrated  combat  air  operations  as  opposed  to  ground  combat,  most  of  the  ROTC 


36 


participants  in  this  study  were  from  the  USAF  ROTC  detachment  (i.e.  Air  Force  -  68 
(73.91%);  Navy  -  16  (17.3%);  Army  -  8  (8.69%)). 

While  undergraduate  psychology  students  received  course  extra  credit  for  their 
participation,  it  was  not  possible  to  provide  similar  rewards  to  the  ROTC  participants.  Due 
to  the  unique  nature  of  the  experimental  simulation  used  in  this  study,  the  detachment 
commanders  believed  the  training  opportunities  inherent  in  the  simulation  used  in  this  study 
were  adequate  compensation  for  the  cadets  and  midshipmen  taking  part  in  the  study.  The 
USAF  ROTC  staff  also  attempted  to  increase  interest  in  the  study  by  incorporating  student 
participation  and  performance  into  their  detachment  competition  for  end  of  the  year 
recognition  awards.  The  award  contribution  from  the  study  was  based  on  the  number  of 
cadets  participating  from  each  cadet  flight.5 

After  discussing  possible  motivational  rewards  with  members  of  the  senior  USAF 
cadet  command  staff,  the  experimenter  also  agreed  to  provide  refreshments  for  a  study  break 
for  the  flight  finishing  with  the  most  points  at  the  end  of  the  experiment.  Each  participant’s 
final  performance  scores  were  recorded  and  summed  according  to  their  flight  assignment  to 
determine  the  winner.  Informal  feedback  from  the  participants  indicated  this  enhanced  their 
willingness  to  participate.  Informal  feedback  also  indicated  that  the  majority  of  ROTC 
participants  were  primarily  motivated  by  a  desire  to  fly  the  simulation. 

Once  individual  difference  measures  were  collected,  participants  were  assigned  to 
one  of  two  flight  teams  (e.g.,  lead  flight  team  and  wing  flight  team)  based  on  pre-determined 
performance  criteria.  To  enhance  the  learning  outcomes  for  their  training,  the  detachment 
commanders  required  that  students  from  the  ROTC  units  organize  their  own  four-person 
teams.  Once  they  arrived  for  their  experimental  session,  ROTC  rank  was  used  as  the  primary 
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criteria  for  deciding  which  students  would  be  pilots.  The  two  highest-ranking 
cadets/midshipmen  were  designated  pilots,  while  the  two  lower  ranking  individuals  were 
designated  as  weapons  specialists  and  randomly  assigned  to  the  two  simulated  aircraft.  The 
highest-ranking  cadet/midshipman  was  also  designated  as  the  formal  leader,  i.e.  mission 
commander,  of  the  team.  If  the  participants  were  of  equal  rank,  then  their  performance  on 
the  psychomotor  test  battery  was  used  to  make  these  determinations  with  the  higher  score 
receiving  the  designation. 

Undergraduate  Psychology  Students.  Forty-eight  (33.3%)  of  the  participants  were 
recruited  from  undergraduate  psychology  courses  following  procedures  established  by  the 
university  and  psychology  department.  As  mentioned  earlier,  subjects  were  randomly 
assigned,  times  permitting,  to  the  three  experimental  conditions,  i.e.  coordination  training, 
strategy  training,  and  control.  Unlike  the  ROTC  participants,  undergraduate  psychology 
students  received  extra  credit  course  hours  (total  =  5)  as  compensation  for  taking  part  in  the 
complete  study. 

Subjects  were  required  to  attend  two  research  sessions.  During  the  first  session, 
extensive  biodata  was  collected  on  the  subjects  for  later  analysis.  Subjects  also  completed 
the  psychomotor  test  battery  mentioned  earlier.  The  session  lasted  about  one  hour,  so 
subjects  received  one  hour  of  extra  credit  as  compensation.  At  the  end  of  this  session, 
subjects  scheduled  their  experimental  session.  Although  encouraged,  subjects  were  not 
required  to  attend  both  sessions  to  receive  any  portion  of  the  total  extra  credit  hours  approved 
for  participating  in  this  study. 

Once  arriving  for  their  experimental  session,  participants  were  assigned  to  one  of  two 
simulated  aircraft,  i.e.  lead  or  wing.  Subjects  scoring  highest  on  the  psychomotor  test  battery 
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were  designated  as  pilots.  Those  with  the  highest  scores  were  designated  as  the  leader,  i.e. 
mission  commander,  of  their  four-person  team.  The  lowest  scoring  individuals  were 
randomly  assigned  as  weapon  specialists.  After  completing  all  experimental  mission 
simulations,  or  when  the  four-hour  time  limit  expired,  subjects  received  an  additional  four 
extra  credit  hours  for  their  participation.  If  time  was  running  short,  teams  were  asked  if  they 
wanted  to  continue  the  experiment  or  wished  to  terminate.  No  experimental  missions  were 
begun  unless  time,  either  defined  by  the  scheduled  limits  or  by  the  team’s  decision  to 
continue,  permitted  their  completion.  Each  of  the  36  teams  participating  in  this  study 
completed  each  of  the  three  experimental  mission  simulations. 

Analysis  of  Potential  Sample  Population  Effects.  As  described  earlier,  the  design  of 
this  study  was  a  2  x  3  x  3  mixed  factorial  design  with  four  dependent  variables  (i.e., 
performance,  interpersonal  process  behaviors,  action  process  behaviors,  and  transition 
process  behaviors).  In  order  to  check  for  any  differences  due  to  subpopulations  (i.e.,  ROTC 
vs.  undergraduate  psychology  students),  a  series  of  subanalyses  was  completed  to  determine 
if  sample  population  was  exerting  any  significance  effects  on  each  of  the  dependent 
variables. 

According  to  these  analyses  (see  Appendix  B,  Table  1),  sample  population  did  not 
significantly  influence  MTS  performance  (i.e.,  ROTC:  mean  =  125.42;  Psyc:  mean  = 

104.17).  Teams  comprised  of  subjects  from  one  population  sample  did  not  perform 
significantly  better  or  worse  than  teams  composed  from  the  other  sample.  The  two  samples 
also  did  not  differ  in  their  interpersonal  process  ratings  (i.e.,  ROTC:  mean  =  3.13;  Psyc: 
mean  =  3.13)  (see  Appendix  B,  Table  2).  However,  as  seen  in  Appendix  B,  Tables  3  and  4, 
population  type  did  significantly  influence  action  and  transition  process  behaviors.  ROTC 
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students  received  higher  ratings  and  scores  on  these  two  variables  (i.e.,  action:  mean  =  3.37; 
transition:  mean  =  .47)  than  their  undergraduate  psychology  student  counterparts  (i.e.,  action: 
mean  =  2.70;  transition:  mean  =  .006).  However,  it  was  decided  that  these  effects  did  not 
require  additional  controls  within  the  later  analyses  since  the  primary  dependent  variable  in 
the  model,  i.e.  MTS  performance,  was  not  significantly  affected  by  population  type. 
Simulation. 

Overview/layout.  Two  pairs  of  subjects,  functioning  as  two  separate  combat 
aircrews,  were  electronically  linked  together  in  an  integrated  multi-team  low-fidelity 
environment.  Although  the  two  personal  computers  (PCs)  had  identical  capabilities,  one 
computer  station  was  designated  as  the  lead  flight,  while  the  other  functioned  as  the  wing 
flight  (see  Appendix  C).  While  each  team  controlled  the  flight  and  operation  of  their  aircraft, 
the  two  teams  were  linked,  both  electronically  and  by  task  requirements,  in  a  multi-team 
operation. 

Based  on  earlier  research  on  task  requirements  associated  with  maintaining  aircraft  in 
flight  (see  Casner,  1994),  feedback  from  an  experienced  USAF  instructor  pilot,  and  SME 
judgments,  mission  task  responsibilities  were  divided  between  the  pilot  and  weapon 
specialist  positions  (see  Table  3).  The  goals  for  this  task  allocation  were  to  reflect  the 
established  division  of  work  required  to  control,  guide,  and  navigate  aircraft  in  flight  (see 
Casner,  1994),  while  creating  interdependency  between  the  two  team  members  within  each 
aircraft.  As  discussed  earlier,  the  lead  aircraft’s  pilot  acted  as  the  designated  mission 
commander  (i.e.,  leader)  for  the  two  teams.  As  such,  he/she  had  final  decision  making  power 
for  resolving  problems  and  achieving  objectives  during  the  simulation. 
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There  was  a  four-way  wooden  partition  separating  team  members  and  teams  (i.e., 
lead  and  wing)  from  one  another.  The  PC  monitor  output  was  split  so  that  each  dyad  (i.e. 
pilot  and  weapons  specialist)  had  access  to  exactly  the  same  visual  information.  The  layout 
also  prevented  a  subject  from  directly  interfering  or  intervening  in  the  actions  taken  by  their 
team  partner  or  the  other  team.  The  teams’  only  link  to  one  another  was  through  verbal 
communication  over  their  headsets. 

Hardware:  Audio.  Each  team  member  had  a  microphone-equipped  headset  to 
communicate  with  their  partner  and  the  other  team  (i.e.  lead  and  wing),  collectively.  When 
activated  by  the  weapons  specialist,  between  team  communications  were  open  to  both  the 
pilot  and  weapons  specialist  positions  simultaneously.  Teams  could  not  control  these 
communications  in  such  a  way  to  conduct  “private”  conversations  between  individuals  (e.g., 
pilot  to  pilot,  weapons  specialist  A  to  pilot  B).  The  weapons  specialist  controlled  the  type  of 
communications  (i.e.  within  or  between  teams)  used  by  their  team  by  activating  a  switch  on  a 
control  box  connecting  the  team  to  their  counterparts. 

Simulated  combat  sounds,  communications  from  computer  controlled  entities  (i.e.,  all 
aircraft  other  than  the  experimental  teams),  and  automated  audio  prompts  and  status  reports 
embedded  in  the  simulation  program  were  presented  in  stereo  in  the  headsets. 
Communications  within  and  between  teams  were  heard  in  mono  channels.  This  permitted 
these  communications  to  be  isolated  to  the  left  earphone,  for  within  team  communications, 
and  the  right,  for  between  team  communications. 

This  audio  set-up  not  only  reflected  some  of  the  very  basic  attributes  of  operational 
systems  presently  in  use  by  airborne  command-and-control  teams,  but  it  provided  the 
subjects  with  an  additional  audio  cue  to  distinguish  the  source  and  identity  of  audio  inputs. 
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This  system  also  enhanced  the  ability  of  SME  evaluators  monitoring  the  teams’  performance 
and  actions  to  locate  the  source  of  contextual  cues  used  by  the  teams  during  the  simulation. 
Subjects  did  not  have  the  ability  to  eliminate  this  “background”  noise  other  than  completely 
removing  their  headsets.  Subjects  were  also  directed  to  conduct  all  conversations  during  the 
experimental  session  through  the  attached  microphones  and  headsets,  thus  permitting  their 
audio  transmissions  to  be  recorded  for  later  analysis. 

Hardware:  Computers.  The  flight  simulation  used  in  this  study  were  run  on  a  100 
Base  TX,  TCI/P  local  area  network  (LAN)  of  Pentium  II 300  PCs.  Each  PC  contained  64MB 
of  RAM  and  a  Direct3D  video  accelerator  with  3  MB  of  video  RAM  running  under  Win95. 
Individual  teams  operated  off  a  single  CPU.  A  video  splitter  was  used  to  transmit  the  video 
signal  from  this  unit  to  the  pilot  and  weapons  specialist  positions.  This  layout  allowed  both 
participants  within  a  single  aircraft  to  view  exactly  the  same  video  displays  at  exactly  the 
same  time.  Both  subject  positions  had  19”  SVGA  monitors  set  at  800X600  resolution  to 
view  the  video  signal.  Additional  similarly  configured  PCs  were  available  on  the  LAN  for 
mission  editing,  recording  flight  information,  and  functioning  as  an  experimenters’  station. 

The  subject  operating  as  the  pilot  used  a  mounted  Thrustmaster  F-22  joystick  and  a 
F-16  Throttle  Quadrant  System  to  control  the  flight  of  the  aircraft.  The  joystick  was 
programmed  to  permit  the  pilot  to  not  only  maneuver  the  aircraft,  but  various  buttons  and 
switches  allowed  him/her  to  alter  the  team’s  cockpit  views,  fire  weapons,  and  activate  air 
brakes.  The  throttle  allowed  the  pilot  to  control  the  simulated  aircraft’s  airspeed. 

The  weapons  specialist  used  a  standard  PS2  style  keyboard  to  perform  his/her  various 
tasks.  Since  time  for  training  was  limited,  approximately  45  minutes  to  an  hour,  several  keys 
were  highlighted  with  color-coded  combinations  and  labels  to  help  the  weapons  specialist 
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carry  out  his/her  tasks  (see  Table  3).  Chief  among  these  tasks  were  selecting  appropriate 
weapon  systems  for  a  selected  target,  creating  “shoot  lists”  for  the  active  weapon  system, 
cycling  targets,  and  aiding  the  pilot’s  efforts  to  avoid  enemy  attacks  by  releasing  chaff  and 
flares  (defensive  “missile  decoys”)  and  controlling  aircraft  electronic  emissions  (i.e., 
EMCON).  Teams  were  taught  that  they  should  maintain  a  low  level  of  electronic  emissions, 
i.e.  remain  stealthy,  while  merely  traveling  along  their  flight  routes.  They  also  understood 
the  importance  of  increasing  their  EMCON  settings  when  it  came  time  to  aggressively  seek 
out  and  destroy  targets. 

Hardware:  Data  Recording.  All  audio  communication  between  and  within  teams,  in 
addition  to  all  game  sounds,  were  routed  from  the  audio  mixer  to  a  stereo  VCR.  By  using  a 
video  splitter,  each  team’s  video  displays  were  also  captured  and  recorded.  This  layout 
provided  experimenters  and  SME  raters  with  an  excellent  method  for  verifying  team  actions 
and  performances  after  mission  completion. 

A  built-in  feature  of  the  simulation  used  in  this  study  is  an  Air  Combat  Maneuvers 
Instrumentation  (ACMI)  program.  This  program  created  a  complete  digital  recording  of  the 
team’s  actions,  in  terms  of  their  assigned  aircraft,  during  an  experimental  mission.  These 
recordings  were  stored  in  a  compressed  digital  file  and  are  available  for  later  review  in  much 
the  same  way  that  one  operates  and  watches  a  VCR  recording.  ACMI  files  can  be  viewed 
from  a  number  of  perspectives  including  a  cockpit  view  (akin  to  what  the  players  actually 
saw  during  the  mission),  a  top-down  satellite  view,  and  even  a  “free”  view  whereby  an 
operator  can  freely  adjust  the  view  to  any  position. 

Experimenters  reviewing  a  team’s  performance  can  toggle  on/off  various  labels 
identifying  each  specific  entity,  weapon  fired,  and  other  pertinent  information.  These  files 
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also  contained  a  running  data  log  of  events  such  as  weapons  fired,  targets  damaged  or 
destroyed,  and  the  exact  flight  path  that  an  aircraft  took.  Due  to  their  high  fidelity,  and  the 
design  of  the  experimental  missions  (discussed  later),  these  files  were  used  to  score  the 
combined  teams’  performance  during  each  experimental  mission. 

Software:  Simulation.  A  low  fidelity,  PC  based  flight  simulator — F22  Total  Air  War 
(TAW)  (Digital  Image  Design  (DID)  Limited,  1998),  was  used  to  create  the  experimental 
conditions  needed  for  this  study.  This  commercially  available  software  contains  a  computer 
generated  simulation  of  the  F-22  aircraft  immersed  within  a  dynamic  combat  environment. 
Unlike  its  current  prototype  counterpart,  the  F-22  aircraft  depicted  in  the  simulation  is 
equally  capable  of  air-to-ground  and  air-to-air  operations.  Although  it  was  possible  to  place 
the  subjects  in  virtually  any  type  of  combat  aircraft,  using  the  F-22  simulation  increased  the 
survivability  of  the  teams  by  providing  additional  protection  through  simulated  stealth 
capabilities.  These  capabilities  generally  reduced  a  simulated  enemy’s  ability  to  detect  and 
target  the  subject  team.  It  was  believed  that  this  capability  was  necessary  to  provide 
participants  with  an  adequate  chance  to  complete  the  experimental  sessions,  given  the  limited 
amount  of  time  available  for  training. 

Over  50  other  types  of  allied  and  enemy  aircraft  were  available  for  any  given  mission 
in  TAW,  and  over  30  types  of  ground  units  and  20  types  of  ships  may  also  be  depicted  in  any 
given  mission  scenario.  Additional  military  aircraft  modeled  in  TAW  included  transports 
(e.g.,  C-130  Hercules;  II-76MF  Candid),  air  refuelers  (e.g.,  KC-135R  Stratotanker;  II II-78M 
Midas),  airborne  command  &  control  (e.g.,  E3  Sentry;  A-50  Mainstay),  reconnaissance 
(944B  Dark  Star,  Mig-25RB  Foxbat),  strike  (e.g.,  Tornado,  Mig-27  Flogger),  fighter  (e.g., 
F15C  Eagle,  SU35  Flanker),  and  helicopters  (e.g.,  AH64  Apache,  Mi-24D  Hind).  The 
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simulation  also  included  civilian  jets  (e.g.,  Lear  Jets,  Boeing  767).  Ground  forces  depicted  in 
the  simulation  included  civilian  (e.g.,  trains;  limousines)  and  military  supply  vehicles  (e.g., 
Humvees,  Bedford  trucks),  light  armor  (e.g.,  M-163  Vulcan,  BMP-3),  heavy  tanks  (e.g.,  Ml 
Abrams,  T-80),  surface-to-air  missile  launchers  (SAMs)  and  anti-aircraft  artillery  (AAA) 
vehicles.  Ships  can  range  from  civilian  (e.g.,  ferries,  oil  tankers),  to  military  light  attack 
(e.g.,  Hovercraft,  gunboats),  heavy  attack  (e.g.,  various  Destroyers  and  Battle  ships),  and 
aircraft  carriers  (e.g.,  Nimitz  class). 

TAW  also  had  the  capability  to  include  up  to  16  countries  (e.g.,  US,  Russia,  Sudan, 
Ethiopia)  during  any  mission  or  campaign  scenario.  While  traditional  NATO  and  other 
alliances  were  available  in  the  simulation,  the  mission  designers  and  experimenters  could 
have  selected  any  other  alliance  combinations.  The  important  consideration  for  subject 
teams  was  that  in  any  given  flight,  it  was  quite  likely  that  multiple  allied,  enemy,  and  neutral 
entities  would  be  present.  This  consideration  limited  the  effectiveness  of  video  gaming 
strategies  relying  on  attacking  and  destroying  everything  the  player  encountered  during  the 
experimental  missions.  Just  as  in  operational  combat  missions,  there  are  many  actors  to 
consider  when  waging  war. 

While  naturally  limited  to  what  can  be  depicted  on  a  PC  screen,  aircraft  flight 
modeling  and  weapon  envelopes  were  generally  consistent  with  publicly  available 
information.  For  example,  all  aircraft  could  only  fly  as  high,  fast,  and  far,  comer  and 
otherwise  maneuver,  carry  weapons,  sustain  damage,  etc.  as  described  about  their  operational 
counterparts  in  the  open  press.  Moreover,  weapons  were  only  as  capable  and  effective  as 
discussed  in  open  sources.  The  actions  of  ground  and  naval  units  are  similarly  well  modeled. 

Software:  Displays.  Although  the  view  could  be  altered  by  the  pilot  to  simulate  a 
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combat  pilot  moving  his/her  head  to  look  around  the  cockpit,  the  team’s  primary  view  was 
looking  straight  out  of  the  aircraft’s  canopy.  Superimposed  on  the  windscreen  was  a  HUD 
that  normally  provided  information  about  the  team’s  current:  1)  altitude;  2)  heading;  3) 
airspeed;  4)  engine  speed;  5)  flight  attitude  (i.e.,  pitch  and  yaw);  6)  gravitational  forces 
experienced;  7)  EMCON  setting;  8)  direction  to  the  next  waypoint;  and  9)  mission  time 
remaining,  along  with  additional  information  depending  on  its  setting.  Three  additional 
HUDs  were  available  to  the  team  and  under  the  weapon  specialist’s  control. 

In  the  air-to-air  (A2A)  and  air-to-ground  (A2G)  modes,  weapons  selection  and 
number  available  were  displayed.  Additionally,  each  mode  presented  the  team  with  valuable 
information  when  air  or  ground  targets  were  selected.  In  the  A2A  mode,  position  and  flight 
information  (i.e.,  speed,  altitude,  heading,  distance,  closing  velocity,  and  aircraft  type)  was 
provided  on  “targeted”  enemy  aircraft.  Notably,  because  of  the  advanced  radar  and 
electronic  identification  capabilities  associated  with  the  F22  in  the  simulation,  it  was  not 
possible  to  lock  A2A  weapons  on  to  allied  aircraft.  The  A2G  mode  also  displayed  critical 
range-to-target  information  about  selected  ground  targets.  Since  ground  vehicles  had  no 
means  of  electronically  identifying  themselves,  teams  had  to  use  visual  displays  in  the 
cockpit  in  conjunction  with  HUD  information  to  avoid  attacking  friendly  forces. 

Finally,  the  navigation  mode  provided  team  members  with  vital  flight  route 
information.  Using  this  HUD,  teams  could  see  if  they  were  off  their  preplanned  flight  route 
in  terms  of  speed,  altitude,  heading,  and  timing.  Although  not  normally  a  factor  during  the 
missions  used  in  this  study,  teams  could  also  view  their  fuel  status.  During  combat,  when  the 
other  HUDs  are  depicting  important  targeting  information,  teams  could  quickly  go  to  this 
HUD  to  determine  their  position  relative  to  their  preplanned  flight  route. 
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After  being  activated  by  the  weapons  specialist,  two  additional  displays  were 
superimposed  on  the  bottom  comers  of  the  HUD.  Both  provided  a  top-down  overview  of  the 
subjects’  position  and  the  objects  around  their  aircraft.  Various  types  of  information  on  both 
displays  were  color  coded  for  consistency  and  to  enhance  their  cue  value.  In  addition,  the 
range  for  both  displays  could  be  adjusted  by  the  weapons  specialist  from  15  to  250  nautical 
miles  in  five  graduated  steps. 

On  the  lower  left  of  the  computer  screen  was  a  display  that  focused  on  targets  and 
threats  to  the  team.  This  display  included  distance  rings  so  that  teams  could  estimate  how  far 
away  any  given  target  was  from  them.  If  a  target  is  “locked-on”  (i.e.,  selected  for  attack)  by 
the  team,  it  was  highlighted.  A  team  could  also  determine  if  any  enemy  vehicle  or  aircraft 
was  attempting  to  lock  on  their  position  using  radar.  In  addition,  incoming  and  outgoing 
missiles,  enemy  and  neutral  aircraft,  and  ground  air  defenses  were  depicted  and  color-coded 
on  this  display. 

On  the  lower  right  of  the  computer  screen  was  a  second  situational  display.  In 
addition  to  the  information  about  the  position  of  objects  in  the  team’s  proximity  in  the  target- 
threat  display,  this  display  included  a  copy  of  the  team’s  preplanned  flight  route.  Key  in  this 
display  were  the  locations  of  geographic  markers,  called  waypoints.  These  waypoints  acted 
as  reference  keys  for  all  the  team’s  required  actions.  Although  radar  signatures  were  not 
displayed  on  this  “aid,”  it  provided  the  team  with  an  important  source  of  information  for 
maintaining  their  flight  course  and  orienting  themselves  to  their  environment. 

In  addition  to  the  various  visual  displays  described  above,  the  TAW  simulation 
provided  various  auditory  cues  to  enhance  the  “realism”  of  the  simulation.  Notably,  the 
game  provided  state-of-the-art  modeling  of  weapons  launches,  explosions,  and  engine  noises. 
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In  addition,  specific  auditory  cues  included  an  electronic  verbal  “shoot”  command  when 
selected  targets  came  within  the  effective  range  of  the  team’s  weapon  system.  Team 
members  also  received  a  verbal  “missile  launch”  warning  when  fired  upon  by  an  enemy 
missile  launcher  or  aircraft.  Simulating  a  radar  attack  warning  alarm  in  real  combat  aircraft, 
a  continual  auditory  warning  sound  was  provided  as  long  as  a  missile  maintained  a  lock  on 
the  team’s  aircraft.  If  hit  by  enemy  fire,  the  subjects  not  only  heard  the  impact,  but  also 
verbal  warnings  of  aircraft  systems  failures. 

Software:  Mission  Scripting.  Weaver,  Bowers,  Salas,  and  Cannon-Bowers  (1995) 
emphasized  that  experimental  team  tasks  must  provide  a  situation  in  which  multiple 
operators  are  required  to  interact  in  an  interdependent  manner.  However,  Kozlowski  and 
Gully  (1996)  insisted  that  typical  laboratory  settings  “...lack  the  complexity,  dynamics,  and 
emergent  characteristics  that  make  NDM  [natural  decision  making]  tasks  so  unusual  and 
challenging”  (p.3).  They  and  other  researchers  (e.g.  Goodwin,  1997;  Mathieu  &  Goodwin, 
1997;  Stout  et  al.,  1990;  Kozlowski  &  Gully,  1996;  Weaver  et  al.,  1995)  have  been 
successful  in  employing  low-fidelity  personal  computer  (PC)  flight  simulations  to  study  team 
behaviors  and  effectiveness. 

Kozlowski  and  Gully  (1996)  contended  that  the  key  to  making  computerized 
simulations  useful  is  their  degree  of  psychological,  not  just  physical,  fidelity  to  the  actual 
field  situation.  They  used  the  term  psychological  fidelity  to  refer  to  “. .  .the  degree  to  which 
key  factors  and  processes  activated  by  the  simulated  situation  mirror  characteristics  in  the 
natural  situation”  (Kozlowski  &  Gully,  1996,  p.9).  While  PC-based  programs  lack  the 
“realism”  present  in  high-fidelity  military  and  commercial  airline  simulators,  they  have 
proven  effective  in  isolating  team  constructs  and  behaviors  using  novice  subject  populations. 
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These  relatively  basic  simulations  have  provided  researchers  the  opportunity  to  employ 
novice  subjects  with  minimum  familiarization  training,  thus  allowing  them  to  maximize  their 
use  of  abundant  student  populations  rather  than  depending  on  more  limited  expert  subject 
populations. 

As  in  other  PC-based  simulations,  the  Air  Campaign  Effectiveness  Simulation 
(ACES)  employed  in  this  study  provided  a  rich  and  dynamic  team  performance  environment. 
For  example,  during  one  mission  employed  in  this  study,  teams  beginning  the  simulation 
were  immediately  confronted  by  potent  enemy  air  defenses  operating  just  beyond  their 
weapons  range.  As  the  teams  adjusted  their  flight  altitude,  in  accordance  with  mission 
guidelines,  they  noted  several  friendly  aircraft  moving  toward  and  engaging  these  enemy 
positions.  Despite  being  sighted  and  “locked  on”  by  the  enemy’s  tracking  radar,  teams  had 
to  ignore  the  more  distant  threat  to  locate  and  identify  a  more  immediate  threat  closer  to  their 
position  but  hidden  from  their  view  by  the  surrounding  terrain. 

Once  this  threat  was  disposed  of,  teams  had  to  find  their  way  to  their  next  assigned 
target’s  location  by  following  a  river  winding  its  way  through  the  area.  Along  the  way,  they 
encountered  many  friendly  ground  forces  and  moved  through  areas  where  aircraft  were 
engaged  in  pitched  combat.  The  teams  also  had  to  avoid  destroying  neutral  passenger  ferries 
being  harassed  by  enemy  hovercraft  and  patrol  boats,  while  destroying  the  enemy  vessels. 

As  they  arrived  in  their  main  target  area,  teams  had  to  distinguish  the  enemy  ground 
forces  from  friendly  ground  forces  moving  to  attack  them.  Despite  many  opportunities  to 
attack  enemy  air  forces  operating  in  the  area,  teams  were  required  to  focus  on  their  assigned 
ground  targets  and  rely  on  computer  controlled  allies  for  their  air  defense.  While  this 
dynamic  environment  presented  the  MTS  with  many  challenges,  it  also  presented 
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experimenters  with  many  unique  challenges  to  create  it. 

Based  on  feedback  from  operational  experts6  and  professional  military  colleges,7 
TAW  provided  an  unique  PC-based  software  platform  for  creating  and  realistic  challenging 
combat  missions.  Thanks  to  the  recommendations  of  the  TAW  software  designers,  an 
editing  program  was  used  to  control  the  exact  conditions  and  events  comprising  each  training 
and  experimental  mission.  The  Air  Defense  Fighter  Graphic  User  Interface  Editor  (AGE), 
version  2.14,  Independent  Mercenary  Corps  (IMC),  software  program  provided  the  capability 
to  script  simulated  mission  parameters  to  the  specifications  required  by  the  experimental 
conditions  employed  in  this  study. 

This  editing  program  allowed  programming  control  over  every  aspect  of  the  battle 
situation  (e.g.,  the  number  and  types  of  allied  and  enemy  aircraft  present  and  their  flight 
paths,  the  number  and  types  of  allied  and  enemy  ground  units  and  their  actions,  waypoint 
placements,  weapons  loads,  etc.).  This  control  was  needed  to  tailor  the  embedded  events 
within  the  experimental  missions  used  to  evaluate  subject  team  performance.  Once  missions 
were  programmed,  they  underwent  extensive  trial  runs  with  research  assistants  and  subject 
volunteers.  Any  flaws  in  the  mission  programming  or  inconsistencies  with  the  experimental 
objectives  for  any  particular  mission  were  immediately  corrected  and  tested. 

Given  that  basic  design,  researchers  then  relaxed  certain  parameters  (e.g.,  the  number 
and  types  of  enemy  fighter  aircraft,  the  number  and  quality  of  enemy  surface-to-air  units)  to 
make  them  playable  by  novice  teams  in  a  15  minute  time  period.  It  was  also  decided  that  the 
experimental  missions  would  focus  on  A2G  operations,  rather  than  attempting  to  control  a 
more  fluid  air  combat  scenario.  Ground  units  were  selected  as  primary  and  secondary  targets 
so  that  they  would  be  easily  distinguishable  by  novice  teams.  To  clarify  performance 
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scoring,  these  selected  targets  could  only  be  destroyed  by  the  participants.  Thanks  to  the 
control  provided  by  the  editing  program,  no  other  computer  controlled  entity  would  engage 
these  targets. 

In  addition  to  training  missions,  three  experimental  missions  were  designed  and 
tested.  These  missions  were  calibrated  to  be  essentially  parallel,  in  that  the  number  and  types 
of  enemy  and  allied  aircraft  and  ground  units  were  held  within  specified  ranges  regardless  of 
experimental  condition  (see  Table  4).  Review  by  TAW  subject  matter  experts  (SMEs)  and 
initial  pilot  testing  suggested  that  these  three  mission  simulations  represented  equally 
challenging  scenarios.  Analysis  of  the  potential  effects  of  mission  order  and  timing 
confirmed  the  SMEs' judgments  that  the  missions  were  generally  equivalent  (see  Table  5). 

A  series  of  one-way  ANOVAs  were  conducted  to  determine  if  the  order  the  mission 
occurred  significantly  influenced  teams’  performances.  As  reported  in  Table  5,  the  order  in 
which  the  three  missions  occurred  did  not  significantly  affect  MTS  performance  (Ridge 
Jumper:  F(2,  33)  =  .88,  n.s.;  Canyon  Sweep:  F(2,  33)  =  1.61,  n.s.;  River  Run:  F(2, 33)  = 

.95,  n.s.).  This  analysis  indicated  that  there  was  no  advantage  in  having  one  mission  versus 
another  at  any  given  time  in  the  experiment.  For  example,  it  didn't  matter  if  teams  flew  the 
River  Run  mission  first,  second,  or  last,  their  performance  scores  were  comparable  across 
time. 

A  similar  analysis  was  also  completed  to  determine  if  the  time,  i.e.,  mission  1, 2,  or  3, 
a  mission  occurred  impacted  the  teams’  performances.  As  indicated  earlier,  this  analysis 
showed  that  mission  timing  did  not  significantly  influence  MTS  performance  (Time  1  (Tl): 
F(2,  33)  =  3.12,  n.s.;  Time  2  (T2):  F(2, 33)  =  .78,  n.s.;  Time  3  (T3):  F(2, 33)  =  .104,  n.s.) 

(see  Table  5).  In  other  words,  there  was  no  advantage  as  to  which  mission  occurred  at  any 
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given  time.  For  example,  it  didn't  matter  if  teams  flew  Ridge  Jumper,  River  Run,  or  Canyon 
Sweep  during  the  first  mission,  their  performance  scores  were  comparable  across  missions. 

In  summary,  the  missions  were  found  to  be  roughly  equivalent  and  comparable  across  the 
experimental  conditions. 

Premeasure  Session. 

Data  were  collected  in  several  stages  during  two  experimental  sessions,  as  mentioned 
earlier.  Together,  the  two  sessions  lasted  about  4.5  to  5.0  hours  and  were  partitioned  into 
five  phases:  (1)  premeasures,  (2)  pre-training  assessment,  (3)  task  training,  (4)  team 
training  intervention,  and  (5)  mission  engagement.  During  the  first  premeasures  session, 
which  lasted  about  one  hour,  participants  completed  a  battery  of  survey  measures  relevant  to 
research  objectives  of  the  overall  laboratory  study.  Information  collected  from  the  subjects 
included  personality  measures,  KSAOs,  background  data,  and  demographics  (i.e.,  sex, 
ethnicity,  academic  year,  GPA,  and  GRE  scores). 

As  discussed  earlier,  participants  also  took  two  specially  adapted  psychomotor  tests  to 
assess  their  capability  to  track  objects  in  a  visual  field  while  manipulating  electronic  controls, 
i.e.  joystick  and  throttle.  Performance  scores  from  single-limb  and  dual-limb  visual  tracking 
tests,  developed  by  USAF  researchers  at  Brooks  Laboratory,  San  Antonio  TX,  were  used  to 
designate  pilots  and  mission  commanders  among  the  participants.  These  tests  were  adapted 
from  the  USAF’s  Basic  Attributes  Tester  (BAT)  programs.  The  BAT  provides  AFROTC  a 
means  of  obtaining  important  selection  data  on  pilot  applicants  early  in  the  selection  process. 
The  BAT  incorporates  computerized  testing  technology  to  measure  psychomotor,  cognitive, 
and  personality  attributes  that  correlate  with  operational  pilot  performance.  (AFROTCI 36-6, 
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25  Sep  95.)  Although  the  order  of  the  tests  in  the  battery  used  in  this  study  could  be  altered, 
all  participants  took  the  tests  in  the  same  order. 

The  first  of  the  two  tests,  Arc  Pursuit,  was  a  single  limb  test  based  on  Melton’s 
(1947)  original  work  for  the  U.S.  Army  Air  Force.  According  to  Chaiken,  Kyllonen,  and 
Tirre  (under  review),  this  test  required  participants  to  use  the  joystick  to  control  the 
movement  of  two  parallel  yellow  lines,  with  five  degrees  of  separation,  within  a  140-degree 
arc-shaped  window.  This  arc  track  measured  four  centimeters  in  height  and  26cm  horizontal 
distance  between  outer  arc  endpoints.  Participants  were  required  to  keep  their  yellow  lines  as 
close  as  possible  to  a  single  red  line  target.  The  target  moved  along  the  arc  at  varying  speeds 
and  reversed  directions  once  it  reached  the  outer  left/right  boundaries  of  the  arc.  Target 
movement  was  determined  by  randomly  combining  seven  rates  of  speed,  ranging  from  35  to 
140  degrees  per  second,  and  three  levels  of  duration,  ranging  from  900  to  1900msec,  with  the 
constraint  that  no  two  consecutive  speeds  were  identical  (Chaiken  et  al.).  Each  trial,  one 
practice  and  three  assessments,  lasted  35  seconds  and  was  followed  with  feedback  detailing 
what  percentage  of  time  the  subject  was  on  or  close  to  the  target.  The  assessment  trial 
performances  were  combined  by  a  programmed  algorithm  to  calculate  a  final  performance 
score  on  the  overall  test.  This  score  was  recorded  for  each  participant  and  was  used  as  a 
secondary  criteria  (i.e.  tie  breaker)  for  designating  pilots  and  mission  commanders.  This  test 
was  programmed  to  appear  first  because  it  required  only  single  limb  coordination  and 
provided  participants,  particularly  those  without  any  experience  using  joysticks  and  throttle 
controls,  with  a  chance  to  familiarize  themselves  with  the  controls  used  in  the  second  test  and 
the  experimental  sessions. 
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The  second  test,  Pop  the  Balloons,  required  subjects  to  move  a  speed  mapped  throttle 
and  joystick  to  control  the  y  and  x  axis  movement,  respectively,  of  a  target  sight  on  the 
computer  screen.  By  maneuvering  the  sight  directly  over  color  designated  target  balloons 
(purple)  and  activating  the  triggering  mechanism  on  the  joystick,  participants  could  pop  the 
balloon  and  score  points.  If  he/she  failed  to  pop  the  targeted  balloon  within  ten  seconds,  it 
turned  red  for  two  seconds  before  it  burst  on  its  own.  Subjects  could  still  gain  credit  for  the 
balloon  by  popping  it  before  it  burst.  The  test  was  set  to  last  four  minutes  without  any 
practice  session.  The  test  began  with  seven  nontarget  balloons  (yellow)  and  one  purple. 

Each  time  a  balloon  was  popped,  one  of  the  remaining  yellow  balloons  turned  purple.  Once 
all  eight  had  been  popped,  eight  more  balloons  appeared.  Feedback  was  given  at  the  end  of 
the  time  session  in  terms  of  number  popped  and  number  timed  out.  These  scores  were 
recorded  for  each  participant. 

Since  this  test  required  subjects  to  manipulate  throttle  and  joystick  controls  to 
maneuver  the  “pin”  over  a  moving  target,  it  was  decided  that  subjects’  scores  on  this  portion 
of  the  test,  i.e.  the  number  balloons  popped,  would  be  the  primary  selection  criteria  for 
designating  pilots  and  mission  commanders,  as  previously  described.  SMEs  indicated  that 
maneuvering  to  destroy  moving  balloons  was  most  analogous  to  the  actions  required  of  pilots 
in  the  simulation.  Since  the  best  performing  pilots  usually  become  mission  commanders,  the 
highest  scorer  among  the  four  team  members  was  designated  the  mission  commander  in  each 
undergraduate  psychology  team.  As  described  earlier,  this  score  was  used  as  a  secondary 
criterion  for  designating  mission  commanders  and  pilots  on  ROTC  teams. 

Once  the  participants  had  completed  all  premeasure  surveys  and  the  psychomotor 
tests,  this  premeasure  session  was  complete.  After  receiving  their  one  hour  of  extra  credit  for 
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participating  in  the  session,  participants  scheduled  themselves  for  the  experimental  sessions. 
Other  than  descriptive  demographic  information  and  using  the  psychomotor  tests  as  selection 
criteria,  the  data  collected  during  the  premeasure  sessions  was  not  used  in  this  study.  It  has 
been  included  in  this  discussion  to  maintain  comparability  with  previously  collected  data  and 
to  provide  a  perspective  of  the  actual  requirements  levied  onto  the  participants. 
Experimental  Sessions  -  Training. 

Task  Training.  Following  recommendations  made  by  Cannon-Bowers  et  al.  (1995), 
team  members  were  guided  through  a  structured  hands-on  task  training  session  based  on  the 
competencies  required  by  the  simulation  (see  Table  6).  Training  focused  on  the  task 
competencies  relevant  to  each  position  (pilot  or  weapon  specialist)  by  an  extensively  trained 
subject  matter  expert  throughout  the  task  training  phase  (see  Table  7).  After  a  general,  ten- 
minute  presentation  reviewing  the  nature  and  purpose  of  the  study,  participants  received 
approximately  35-45  minutes  of  initial  task  training  prior  to  the  training  intervention  and 
participating  in  scripted  combat  missions  (see  Appendix  D). 

These  initial  training  sessions  focused  on  ensuring  the  subjects  were  adequately 
proficient  in  performing  their  individual  task  responsibilities  (e.g.,  flying  and  maneuvering 
the  plane,  weapon  selection,  locking  onto  enemy  targets,  firing  at  enemy  targets,  etc.)  while 
becoming  familiar  with  their  partner’s  responsibilities  (see  Table  3).  Due  to  the  nature  of  the 
experimental  missions,  instruction  concentrated  on  flight  operations  and  the  air-to-ground 
task  areas  while  ensuring  team  members  had  a  basic  mastery  of  the  air-to-air  combat  tasks. 

In  the  final  portion  of  their  training,  participants  completed  a  joint  (i.e.,  all  four  team 
members)  practice  session  which  emulated  the  general  task  and  teamwork  requirements 
existing  in  the  experimental  mission  scenarios  (see  Table  6).  As  in  the  experimental 
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missions,  the  two  dyads  (i.e.,  lead  and  wing  aircraft)  had  to  work  together  to  achieve 
specified  mission  goals  in  the  multi-team  ACES  simulation.  This  set-up  reflected  the  two- 
ship  (i.e.,  two  aircraft)  formation  normally  characterizing  minimum  USAF  requirements  for 
tactical  air  combat  operations. 

Just  as  during  the  experimental  sessions,  teams  received  a  printed  copy  of  their 
specific  mission  objectives,  pictures  of  their  assigned  targets,  and  a  map  of  their  assigned 
flight  route  to  reference  during  the  practice  mission  simulation.  Additionally,  pictures 
identifying  the  various  vehicles  and  aircraft  used  in  the  simulation  were  posted  at  each 
subject  station.  Unique  HUD  references  were  displayed  at  each  experimental  station  for  the 
subject’s  easy  reference  during  the  simulation. 

The  task  training  session  contained  built-in  competency  checks  to  ensure  participants 
were  able  to  carry  out  the  tasks  required  of  them  in  the  simulation.  During  each  phase  of  the 
task  training  session,  the  assigned  SME  instructor  assessed  his/her  participants’  performance 
in  each  competency  area  (Table  7).  Based  on  these  observations,  the  trainer  determined  if 
the  participants  had  acquired  the  necessary  task  competencies  to  continue  in  the  experiment. 
Any  knowledge  and  skill  deficiencies  were  immediately  addressed  by  the  trainer  (e.g.,  verbal 
explanation).  Following  this  retraining,  subjects  could  be  given  a  second  opportunity  to  fly  a 
portion  of  the  joint  practice  mission,  if  time  permitted. 

While  this  procedure  implied  that  some  subjects  may  have  received  more  training 
time  than  others,  I  do  not  believe  this  provided  them  any  advantage  in  the  experimental 
sessions.  Teams  and  the  individuals  composing  them  were  trained  to  achieve  equal  levels  of 
task  competency  before  continuing  in  the  experiment.  Based  on  such  an  approach  (cf., 
Cannon-Bowers  et  al.,  1995;  Goodwin,  1997;  Heffner,  1996;  Salas  et  al.,  1992),  it  is  not 
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necessary  to  duplicate  identical  training  conditions  for  each  individual  to  achieve  equal  levels 
of  competency  in  the  tasks  required  by  the  experimental  simulations. 

Team  Training  Manipulation.  In  the  team  training  intervention  phase  of  this  study, 
team  members  were  guided  through  one  of  three,  structured  team  training  sessions:  strategy 
training,  coordination  training,  or  a  control  session.  Each  training  condition  employed  a 
twenty-minute  presentation  supervised  by  an  extensively  trained  SME.  Each  presentation 
reviewed  general  team  mission  responsibilities  and  basic  team  processes  (e.g.,  mission 
analysis,  strategy  formulation/planning,  goal  specification  and  formulation,  systems 
monitoring,  timing/coordination,  etc.)  as  described  in  Marks  et  al.’s  (in  preparation) 
taxonomy.  Each  presentation  also  followed  the  same  basic  format. 

Initially  subjects  were  presented  with  an  overview  of  the  concepts  covered  by  the 
presentation.  Then,  using  a  mixture  of  bulleted  slides  and  recorded  video  segments,  each  of 
the  process  behaviors  described  by  Marks  et  al.  (in  preparation)  was  defined  and  discussed. 
Following  specially  tailored  presentation  scripts  designed  to  limit  the  concepts  covered  in  the 
training  condition,  experienced  SMEs  posed  as  team  members  modeling  the  behaviors 
described  in  the  presentation.  Each  presentation  ended  with  a  summary,  on  slides  and  in 
audio,  of  the  main  concepts  covered  during  the  training  session.  To  reinforce  these  points, 
subjects  were  verbally  questioned  about  the  central  characteristics  of  the  process  behaviors 
described  and  demonstrated  by  the  presentation. 

Specifically,  the  strategy  training  presentation  reviewed  each  of  the  transition  process 
behaviors  discussed  by  Marks  et  al.,  i.e.  goal  specification,  mission  analysis,  and  strategy 
planning  and  development.  By  the  end  of  the  training  session,  each  subject  should  be  able  to 
identify  their  primary  and  secondary  objectives  (goal  specification),  understand  why  they 
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need  an  appreciation  of  the  other  operations  going  on  around  them  (mission  analysis), 
develop  a  set  of  primary  and  contingency  plans  to  accomplish  their  assigned  mission 
objectives  (strategy  formation  and  planning).  Following  training,  students  responded  in 
writing  to  an  open-ended  question  designed  to  assess  their  knowledge  of  the  concepts 
described  in  the  session.  Additional  questions  also  assessed  their  knowledge  of  those 
concepts  reviewed  in  the  other  training  conditions.  These  responses  were  later  analyzed  for 
their  accuracy  and  content. 

The  coordination  training  presentation  reviewed  each  of  the  action  process  behaviors 
discussed  by  Marks  et  al.,  i.e.  checking  progress  toward  objectives,  monitoring  their 
environment  (both  inside  and  outside  of  their  combat  aircraft),  backing  up  teammates,  and 
coordinating  actions  between  teammates.  This  training  was  designed  to  review  those 
behaviors  required  for  a  single  dyadic  team  (i.e.,  a  pilot  and  weapon  specialist)  to  coordinate 
their  actions.  By  the  end  of  the  training  session,  each  subject  should  be  able  to  demonstrate 
each  of  these  behaviors  while  flying  their  assigned  missions.  As  in  the  strategy  training 
condition,  students  responded  in  writing  to  open-ended  questions  designed  to  assess  their 
knowledge  of  the  concepts  described  in  their  assigned  training  session  as  well  as  their 
knowledge  of  those  reviewed  in  the  other  training  conditions.  These  responses  were  later 
analyzed  for  their  accuracy  and  content. 

Flight  teams  in  the  control  condition  did  not  receive  any  type  of  team  process  training 
intervention.  Subjects  in  this  group  viewed  a  twenty  minute  multimedia  presentation 
consisting  of  scenes  from  the  “Canopy’s  Up”  and  “Frontline  Pilots”  film  series  produced  by 
Blue  Heron  Films.  Although  informative  about  the  general  nature  of  air  combat,  stealth 
technology,  the  F-22,  and  the  special  effects  used  in  the  TAW  simulation  to  mimic 
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explosions  and  weapons  firing,  the  videos  did  not  contain  any  performance  enhancing  or 
instructional  material  relevant  to  the  experimental  conditions.  As  in  the  experimental  teams, 
control  subjects  responded  to  open-ended  questions  that  provided  a  manipulation  check  for 
the  training  interventions  used  in  this  study. 

Following  training  and  prior  to  beginning  the  practice  MTS  mission,  participants 
were  also  tested  about  the  task  material  presented  to  them.  Included  in  their  participant 
packages  were  ten  multiple-choice  questions  assessing  their  knowledge  of  critical  HUD 
information  (i.e.,  task  knowledge).  These  questions  provided  a  measure  of  the  subjects'  task 
competency  and  knowledge.  Their  responses  were  compared  to  a  master  test  key  and  scored 
for  later  analysis. 

As  mentioned  earlier,  subjects'  responses  to  the  three  open  ended  questions  served  as 
manipulation  checks  for  the  training  intervention  conditions  by  querying  them  about  key 
concepts  reviewed  in  each  of  the  training  conditions.  Using  a  point  assessment  derived  from 
SME  responses  and  a  review  of  test  subject  responses,  two  SMEs  rated  the  quality  and 
accuracy  of  each  participant’s  answers  (see  Appendix  E).  Analysis  showed  that  the  SMEs’ 
ratings  were  significantly  correlated  with  each  other  (coordination:  .65,  p<.01;  strategy:  .77, 
p<.01)  ( see  Table  8).  This  indicated  that  the  raters  applied  similar  standards  to  the 
participants’  responses  and  provided  the  justification  for  aggregating  the  two  ratings  into  a 
single  measure  for  later  analyses. 

As  expected,  analysis  of  variance  and  contrast  analysis  indicated  that  trained  teams 
did  significantly  better  than  teams  in  the  control  condition  on  the  knowledge  question 
corresponding  to  their  training  condition  (see  Table  9).  Additionally,  these  teams  also 
performed  better  on  their  corresponding  test  item  than  teams  in  the  remaining  training 
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condition  (see  Table  9).  Simply  put,  subjects  receiving  coordination  training  did  better  on 
the  question  focusing  on  coordination  behaviors  than  subjects  in  either  the  control  or  strategy 
training  condition.  The  same  held  true  for  subjects  in  the  strategy  condition  relative  to  the 
question  requiring  subjects  to  list  important  factors  in  developing  strategies  and  plans.  This 
analysis  also  indicated  that  task  knowledge  was  unaffected  by  training  condition  (Table  9). 

In  other  words,  none  of  the  training  conditions  either  enhanced  or  degraded  the  subjects'  task 
knowledge.  Based  on  this  analysis,  the  training  manipulation  was  successful  in  presenting 
training  most  appropriate  for  one  condition  over  another  or  the  control  condition. 

Complexity  Manipulation.  As  described  earlier,  this  study  employed  a  2  x  3  x  3 
mixed  factorial  design  crossing  environmental  complexity  (high  vs.  low)  with  training 
conditions  (strategy,  coordination,  control)  over  three  experimental  sessions.  Following 
Wood’s  (1986)  research  findings  and  conclusions,  higher  levels  of  complexity  were  created 
by  increasing  the  component  (i.e.,  features)  and  dynamic  (i.e.,  predictability)  complexity  of 
the  experimental  sessions.  Specifically,  in  high  complexity  conditions  the  variety  of  target 
objects  at  each  designated  location  was  increased,  thus  increasing  the  number  of  target 
features  that  the  teams  must  distinguish  to  accomplish  their  assigned  tasks  (see  Table  4). 

For  example,  during  one  mission  simulation,  a  team  had  to  discern  the  differences 
between  two  types  of  boats  -  one  enemy  and  one  neutral.  However,  in  the  high  complexity 
version  of  this  same  mission,  participants  had  to  distinguish  between  three  types  of  boats  - 
two  enemy  and  one  neutral.  The  important  consideration  in  this  problem  was  that  only  one 
type  of  enemy  ship  was  assigned  as  either  a  primary  or  secondary  target.  As  found  in  related 
cognitive  research  (see  Treisman,  1991),  this  increased  the  complexity  of  the  task  for  the 
teams  by  requiring  them  to  distinguish  between  multiple  features  of  the  displayed  targets 
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before  taking  action. 

In  the  high  complexity  condition,  teams  also  received  less  specific,  more  ambiguous 
information  on  the  movement  and  location  of  potential  target  objects  and  threats  than  in  the 
low  complexity  condition.  In  the  low  complexity  condition,  enemy  and  friendly  forces  were 
specifically  identified  by  type,  e.g.,  T-80  tanks  and  SU-27s.  These  reports  also  indicated 
where  these  forces  were  located  relative  to  the  pre-planned  flight  plan  and  waypoints  (i.e., 
geographical  markers  on  a  map)  in  addition  to  more  general  grid  coordinates.  In  contrast,  the 
high  complexity  condition  referenced  objects  in  the  mission  area  in  more  general  terms  -  as 
friendly  or  enemy  forces.  Their  locations  were  also  described  in  terms  of  their  general  grid 
coordinates,  which  provided  a  much  larger  area  for  the  team  to  potentially  search  for  their 
targets.  In  this  manner,  teams  in  the  high  complexity  condition  were  less  able  to  predict  the 
locations  and  possible  actions  of  enemy  forces  than  they  could  in  the  low  complexity 
condition. 

Embedded  in  the  participants’  experimental  packets  were  several  items,  scaled  one  to 
five  (ranging  from  strongly  disagree  to  strongly  agree),  designed  to  assess  subjects’ 
individual  perceptions  of  the  complexity  of  their  mission  and  task  environment.  Analysis  of 
these  five  items  indicated  that  four,  i.e.  items  10, 12, 13,  and  14,  were  reliable  and  consistent 
indicators  of  the  subjects'  perceptions  of  the  complexity  of  the  environment  in  which  they 
were  immersed  (Tl:  a  =  .66;  T2:  a  =  .60,  T3:  a  =  .58).  Item  10  indicated  the  subjects  had  to 
accomplish  an  unusually  large  number  of  tasks  to  complete  the  mission.  Item  12  stated  that 
the  subjects  were  given  accurate  information  to  base  their  plans  upon.  Item  13  indicated  that 
the  mission  was  predictable  and  required  only  small  adjustments  to  their  plans.  Finally,  item 
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14  stated  that  the  mission  was  easy  and  straightforward.  Aggregating  these  responses 
provided  a  indicator  of  how  complex  the  subjects  perceived  their  task  environment. 

An  ANOVA  examined  the  subjects'  perceptions  of  environmental  complexity  over 
time.  Based  on  comparisons  of  their  means  across  time,  this  analysis  indicated  that  subjects 
failed  to  perceive  any  significant  difference  between  the  two  complexity  conditions  (means: 
Low  (Tl)  =  2.92,  High  (Tl)  =  2.93;  Low  (T2)  =  2.91,  High  (T2)  =  2.82;  Low  (T3)  =  2.93, 
High  (T3)  =  2.95).  The  analysis  also  indicated  that  only  a  three-way  interaction  between 
time*complexity*training  was  significantly  related  to  subjects'  perceptions  of  the 
environment's  complexity  (see  Table  10).  This  would  seem  to  indicate  that  subjects' 
perceptions  were  more  of  a  function  of  which  training  condition  they  were  assigned  than  the 
level  of  complexity.  In  short,  these  analyses  indicate  that  the  manipulations  used  in  this 
study  failed  to  affect  subjects'  perceptions  of  their  environment's  complexity. 

It  is  important  to  keep  in  mind  that  subjects  only  experienced  one  level  of  complexity 
(i.e.,  either  high  or  low)  throughout  their  experimental  session.  Based  on  previous  research 
(c.g.,  Hackman  &  Morris,  1975;  Salas  et  al.,  1992;  Saavedra  et  al.,  1993;  Wood,  1986)  and 
the  number  of  entities  present  in  the  subjects'  perceptual  field  (see  Table  4),  the 
manipulations  used  in  this  study  adjusted  environmental  complexity  more  in  terms  of  degree 
(i.e.,  complex  vs.  overload)  than  in  more  absolute  terms  (i.e.,  mundane  vs.  enriched 
environments).  The  failure  of  these  manipulations  could  indicate  that  a  greater  difference 
between  the  two  conditions  was  required  before  subjects  could  perceive  any  significant 
differences.  The  addition  of  eight  distracters  (see  Table  4)  may  not  have  been  sufficient  to 
alter  the  subjects'  perceptions. 
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Experimental  Sessions  -  Missions. 


Teams  began  the  final  phase,  the  mission  engagement  phase,  with  an  in-basket 
exercise  and  mission  planning  session.  The  purpose  of  these  materials,  as  described  in 
Chapter  4,  was  to  assess  important  team  process  behaviors  while  providing  teams  with  the 
information  they  needed  to  complete  each  mission.  Once  they  had  received  their  planning 
materials,  teams  were  given  ten  minutes  to  plan  their  mission  and  select  their  weapons 
options  (described  in  detail  in  Chapter  4).  At  the  end  of  their  planning  session,  the  mission 
commander  was  interviewed  to  provide  a  summation  of  the  team’s  plans  and  reasoning. 

Once  this  was  completed,  team  members  completed  a  series  of  short  surveys  before  entering 
the  experimental  simulation.  Team  members  were  allowed  to  take  important  reference 
materials,  i.e.  printed  flight  map,  pictures  of  assigned  targets,  mission  brief  detailing  assigned 
mission  objectives  and  flight  requirements,  into  the  simulation  for  later  use. 

As  discussed  earlier,  teams  “flew”  three  different  air-to-ground  combat  missions 
following  completion  of  their  training.  Although  these  missions  differed  in  terms  of  terrain 
and  specific  mission  objectives,  they  required  the  same  task  competencies  and  contained  an 
equal  number  of  potential  ground  targets  (i.e.,  primary,  secondary,  and  nonassigned)  (see 
Table  4).  Previous  research  employing  an  event-based  strategy  has  well  established  the 
importance  to  hypothesis  testing  of  standardizing  the  levels  of  performance  across 
performance  task  conditions  (see  Dwyer,  Fowlkes,  Oser,  Salas,  &  Lane,  1997;  Fowlkes, 

Lane,  Salas,  Franz,  &  Oser,  1994).  As  can  be  seen  in  Table  4,  experimenters  designed  the 
missions  to  be  equivalent  in  terms  of  their  mission  objectives,  i.e.  assigned  targets. 

Each  mission  lasted  15  minutes,  or  until  both  aircraft  were  destroyed.  If  one  aircraft 
was  destroyed  before  the  other,  the  “deceased”  team  remained  at  their  station  while  their 
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partners  continued  the  mission.  The  order  of  the  experimental  missions  was  counterbalanced 
to  minimize  any  ordering  effects  on  team  performance  measures.  Although  the 
experimenters  attempted  to  achieve  a  complete  counterbalanced  design,  it  was  not  possible 
due  to  the  problems  in  subject  recruitment  discussed  earlier  (see  Table  11).  However,  as 
discussed  earlier  in  this  paper,  neither  mission  order  nor  timing  had  any  significant  impact  on 
MTS  performance  (see  Table  5),  so  there  was  no  order  effect  requiring  counterbalancing  in 
the  design  and  analyses. 

Once  the  mission  was  completed,  the  MTS  was  debriefed  on  their  mission 
performance  in  terms  of  overall  score,  mission  objectives  achieved,  number  of  assigned 
targets  destroyed  and,  if  required,  the  number  of  neutrals  destroyed.  Team  members  were 
directed  to  complete  a  series  of  post-mission  surveys  before  receiving  the  in-basket  and 
mission  planning  materials  for  the  next  experimental  session.  Their  annotated  maps  and 
mission  briefs  were  collected  and  labeled  for  later  analysis.  If  additional  missions  remained, 
the  team  received  the  planning  materials  for  the  next  mission.  Once  the  planning  session  and 
interview  were  completed,  the  MTS  returned  to  the  experimental  simulation. 
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Chapter  4 
MEASURES 

Multiple  measures  of  team  process  and  performance  effectiveness  were  collected  to 
accurately  evaluate  subjects’  performance  and  provide  the  basis  for  later  analysis.  The 
dimensions  described  in  Marks  et  al.’s  (in  preparation)  taxonomy  were  operationalized  to 
reflect  the  tasks  and  behaviors  required  by  the  simulation  (see  Table  12).  These  operational 
definitions  guided  the  collection  of  data  during  the  experiment  and  provided  a  sound 
foundation  for  analyzing  the  variables  tested  by  the  proposed  model.  They  also  provided  the 
foundation  for  training  SMEs  on  how  to  classify  their  observations  and  understand  what 
particular  observed  behaviors  contribute  to  different  process  and  performance  ratings,  as 
described  by  Marks  et  al. 

As  will  be  described  in  detail  later,  scoring  totals  were  computed  based  on  the 
number  of  primary  and  secondary  targets  (i.e.,  mission  objectives)  destroyed  by  the  MTS. 
These  measures  were  used  as  objective  indicators  of  the  MTS's  performance.  SME  ratings 
and  measures  of  team  process  were  central  in  testing  the  hypotheses  proposed  in  this  study 
(see  Appendix  F).  Additional  measures  were  collected  (e.g.,  teams’  affective  reactions  to  the 
task,  situation,  and  their  performance,  mental  models,  and  shared  mental  models)  as  part  of 
the  project’s  larger  data  collection  effort  and  research  objectives.  These  data  were  not  used 
to  test  any  hypotheses  proposed  in  this  study. 

This  study  employed  a  repeated  measures  design  to  increase  the  number  of  available 
observations  per  team  and  the  power  of  the  analysis  due  to  a  relatively  limited  number  of 
available  subjects  (c.g.,  Cobb,  1998;  Gully,  1994).  SMEs  rated  team  process  behaviors 
evidenced  in  one  of  the  two  dyads  comprising  the  larger  MTS  for  each  of  the  three  missions. 
Following  Dickinson  and  McIntyre's  (1997)  suggestions,  raters  used  a  behaviorally  based, 
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BARS  type  scale  to  rate  the  team  leader,  each  team  member,  and  the  overall  two-person  team 
on  a  five-point  scale  for  each  team  process  dimension  described  by  Marks  et  al.  (in 
preparation)  (see  Appendix  F). 

Team  Process  Measures  -  Transition  Process  Behaviors. 

Marks  et  al.  (in  preparation)  described  transition  processes  as  being  characterized  by 
behaviors  directed  at  transforming  work  units  and  effectively  responding  to  changes  in  the 
work  environment.  According  to  their  description,  transaction  processes  include  mission 
analysis,  goal  specification  and  formulation,  and  strategy  formation.  Following  Brannick  et 
al.'s  (1995)  assertion  that  multiple  observations  were  necessary  to  fully  assess  team  and  task 
process  behaviors,  several  different  measures  were  embedded  in  this  experimental  design  to 
assess  the  quality  of  the  MTS’s  mission  analysis,  goal  specification,  and  strategy  formation 
(i.e.,  transaction  process  behaviors). 

O-Sort.  As  previously  mentioned  in  Chapter  3,  the  in-basket  exercise  consisted  of 
giving  each  four-person  team  a  variety  of  reports  containing  information  both  relevant  and 
irrelevant  to  the  team’s  assigned  mission.  The  belief  was  that  performance  on  this  exercise 
would  be  indicative  of  the  extent  to  which  a  MTS  conducted  a  quality  mission  analysis. 
Reflecting  the  style  and  content  of  operational  military  reports,  situational  reports  (SITREPs) 
contained  information  about  the  position,  strength,  and  movement  of  friendly  forces. 
Intelligence  reports  (INTEL)  provided  information  about  the  position,  strength,  and 
movement  of  enemy  and  neutral  forces  in  the  team’s  mission  area.  There  were  five  specially 
tailored  SITREPs  and  five  INTEL  reports  for  each  mission.  Each  of  these  items  was 
previously  rated  by  SMEs  responsible  for  designing  the  experimental  simulations  for  their 
usefulness/relevance  to  specified  mission  objectives  (see  Appendix  G). 
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Participants  were  directed  to  sort  the  in-basket  materials  in  terms  of  their  importance, 
either  very  important,  helpful,  or  irrelevant  (i.e.  useless)  to  their  assigned  mission.  They 
were  also  instructed  to  accomplish  this  sorting  as  part  of  their  mission  planning.  Each  four- 
person  team  was  given  ten  minutes  to  plan  their  mission  and  strategy,  prioritize  their  goals 
and  mission  objectives,  and  refine  their  roles  and  responsibilities. 

The  subjects’  final  sort  of  the  exercise  materials,  prior  to  entering  the  mission 
simulation,  was  recorded  for  comparison  with  SME  derived  rankings  to  determine  their 
exercise  score.  Subjects  received  two  points  if  they  placed  the  report  in  the  same  stack  as  the 
a  priori  selection.  If  they  were  one  position  off,  e.g.  placing  a  report  in  the  helpful  stack 
when  it  should  have  been  seen  as  irrelevant,  they  received  one  point.  If  they  were  two  places 
off,  e.g.  treating  an  irrelevant  piece  of  information  as  very  important,  the  team  received  no 
points.  This  scoring  method  provided  a  potential  range  of  two  to  twenty  points  for  each 
mission.  Since  the  entire  four-person  team  participated  in  the  sorting,  these  points  were 
totaled  and  the  score  recorded  for  analysis  at  the  MTS  level. 

Weapons  Option  Selection.  Embedded  in  the  teams’  planning  session  and  activities 
was  the  requirement  for  the  team  to  jointly  decide  what  weapons  option  to  select.  These 
options  specified  differing  numbers  of  the  air-to-ground  weapons  available  for  the  team’s  use 
in  the  simulation  (see  Appendix  H).  Since  the  selection  of  weapons  packages  largely 
depended  on  the  team's  estimation  of  enemy  threats  and  target  requirements,  this  selection 
was  believed  to  reflect  the  MTS's  strategy  development.  As  seen  in  Appendix  H,  teams 
could  choose  from  three  separate  weapons  options  to  arm  both  their  aircraft  (i.e.,  both  aircraft 
were  given  the  exact  same  number  and  type  of  weapons).  While  these  options  were  the  same 
for  each  experimental  mission,  there  were  differences  in  how  effective  each  option  would  be 
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in  any  particular  mission. 

SMEs  responsible  for  designing  the  mission  simulations  previously  determined  which 
option  provided  the  most  optimal  weapons  capability  for  each  mission  (see  Appendix  H).  As 
in  the  in-basket  q-sort,  the  weapons  option  selected  by  the  MTS  was  noted  and  later 
compared  with  the  SMEs’  a  priori  rankings.  Selecting  the  most  appropriate  option  gained  the 
team  two  points,  the  next  most  appropriate  option  resulted  in  a  score  of  one,  and  the  least  was 
given  a  zero.  Since  the  entire  four-person  team  participated  in  the  selection  decision,  the 
points  were  totaled  and  the  score  recorded  for  analysis  at  the  MTS  level. 

Interview.  Once  the  planning  session  was  completed  and  the  team’s  q-sort  collected 
for  later  analysis,  the  mission  commander  was  taken  to  a  separate  location  to  be  interviewed. 
During  this  interview,  he/she  was  asked  several  questions  designed  to  provide  a  summation 
of  the  team’s  plans  and  strategy  development  (see  Appendix  I).  The  interview  was  recorded 
on  audio  tape  and  later  evaluated  by  a  SME  who  had  not  previously  interviewed  the  subjects 
they  rated  nor  acted  as  one  of  the  primary  instructors  for  the  mission  commander’s  MTS.  As 
with  other  process  ratings,  behaviorally  based  BARS  ratings  scales  were  developed  to  assess 
the  mission  commander’s  interview  responses  (see  Appendix  J)  in  terms  of  the  transaction 
phase  behaviors  described  by  Marks  et  al.  (in  preparation)  (c.g.,  Dickinson  &  McIntyre, 

1997). 

Sixteen  of  the  168  mission  interviews  conducted  during  the  study  were  randomly 
selected  and  rated  by  each  of  the  three  interview  raters.  The  interrater  correlations  for  the 
two  sets  of  ratings  were  sufficiently  high  enough  across  each  of  the  three  transition  process 
dimensions  (goal  specification:  r  =  .79,  mission  analysis:  r  =  .87,  strategy  formulation:  r  = 

.84)  to  permit  viewing  the  SME  ratings  as  equivalent.  Therefore,  as  described  earlier,  only 
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one  rater  was  used  for  each  MTS  participating  in  the  experiment.  Analysis  indicated  that  the 
interview  ratings  were  consistent  measures  over  time  (T1 :  a  =  .91 ;  T2:  a  =  .90;  T3 :  a  =  .94). 

Summary  Scores.  The  three  methods  of  measurement,  just  described,  provided  two 
assessments  of  mission  analysis  and  strategy  formation,  and  one  (i.e.,  the  interview)  score  for 
goal  specification.  This  was  deemed  to  be  a  balanced  measure  of  transition  process,  since  the 
goals  were  specified  in  detail  in  the  teams'  briefings  for  each  mission.  Since  each  of  these 
three  transition  process  measures,  i.e.  q-sort,  weapons  option  selection,  and  interview,  used 
different  scale  ranges,  it  was  necessary  to  standardize  them  in  order  to  derive  a  composite 
score  for  transition  process. 

Analysis  indicated  that  the  three  transition  process  measures  were  significantly 
related  to  one  another  (see  Table  13).  According  to  Table  13,  the  correlations  between  the 
mission  analysis  rating  from  the  interview  was  significantly  correlated  to  the  q-sort  results. 
This  provided  evidence  that  the  different  measures  were  tapping  into  the  same  domain  — 
mission  analysis.  The  intercorrelations  between  the  ratings  for  strategy  formation,  weapons 
selections,  and  the  q-sort  also  provide  evidence  that  the  different  measures  were  sampling  the 
MTS's  strategy  formation  behaviors. 

Calculating  Z-scores  for  each  measure  provided  the  means  to  compare  and  combine 
these  scores  across  measures  with  different  metrics  (Cohen  &  Cohen,  1983).  This  linear 
transformation,  and  their  significant  correlation  with  each  other,  also  permitted  the  scores  to 
be  combined  into  a  single  summary  score  for  later  analysis.  The  correlations  noted  in  Table 
13  also  indicated  that  any  composite  of  these  measures  could  potentially  be  limited  by 
possible  multidimensionality  between  the  two  factors  highlighted  by  the  correlations,  i.e. 
mission  analysis  and  strategy  formulation.  Analysis  indicated  that  these  composite  scores 
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were  reliable  measures  of  transition  process  behaviors  over  time  (T1 :  a  =  .69;  T2:  a  =  .70; 
T3:  a  =  .80). 

Team  Process  Measures  -  Action  and  Interpersonal  Process  Behaviors. 

As  discussed  earlier,  action  processes  involve  team  member  behaviors  associated 
with  teams  actually  carrying  out  their  plans  and  performance  tasks.  According  to  Marks  et 
al.  (in  preparation),  these  processes  include  monitoring  progress  toward  goals,  systems 
monitoring  (i.e.,  monitoring  internal  technology  systems  and  the  external  environmental  for 
changes  and  task  cues),  team  monitoring  and  backup  behavior,  coordination  activities,  and 
communications  between  team  members.  Interpersonal  processes,  on  the  other  hand,  focus 
on  those  affect  driven  behaviors  associated  with  reducing  interpersonal  conflict  and  keeping 
team  members  motivated  (Marks  et  al.,  in  preparation). 

Following  Dickinson  and  McIntyre’s  (1997)  recommendations,  behavior  based 
BARS  ratings  scales  were  developed  to  assess  each  of  the  two-person  teams  within  the  MTS 
in  terms  of  the  action  and  interpersonal  process  behaviors  described  by  Marks  et  al.  (in 
preparation)  (see  Appendix  F).  The  MTS  instructor  for  each  two-person  team  rated  the 
effectiveness  of  his/her  team  members’  behaviors  in  each  of  the  action  and  interpersonal 
dimensions  described  in  Appendix  F.  These  ratings  employed  a  five  point  scale,  ranging 
from  hardly  any  skill  (i.e.,  1)  to  complete  skill  (i.e.,  5). 

Unrestricted  maximum  likelihood  analysis  indicated  that  MTS  instructors/raters  were 
consistently  able  to  reliably  discern  and  assess  the  two  process  dimensions  over  time  and 
repeated  trials  (see  Table  14).  Since  these  ratings  were  based  on  observations  of  the  two- 
person  teams  (i.e.,  lead  and  wing)  comprising  the  larger  MTS,  it  was  necessary  to  combine 
these  ratings  into  a  summary  rating  for  the  four-person  MTS. 
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Although  the  correlation  between  the  lead  and  wing  dyads  on  the  interpersonal  BARS 
ratings  were  very  limited  over  time  (see  Table  15),  analysis  indicated  that  the  overall 
correlations  for  the  two  dyads  were  moderately  significant  (interpersonal  process  =  .20,  p  < 
.05;  action  process:  .45,  p  <  .01)  (see  Table  15).  Although  these  modest  correlations  indicate 
that  that  a  composite  of  the  two  measures  could  suffer  from  relative  low  construct  validity 
and  internal  consistency,  it  was  determined  that  the  correlations  still  permitted  the  means  of 
the  lead's  and  wing's  interpersonal  and  action  process  behaviors  ratings  to  serve  as  summary 
scores  for  the  larger  MTS.  As  described  earlier,  a  composite  score  combining  the  teams' 
standardized  q-sort  score,  weapons  selections  score,  and  interview  ratings  served  as  the 
measure  of  MTS  transition  process  behaviors  for  later  analyses. 

Team  Performance  Measures. 

As  mentioned  earlier,  the  existing  TAW  programs  were  altered  to  ensure  teams  were 
required  to  plan  and  follow  strategies  designed  to  accomplish  assigned  objectives  to  be 
successful.  The  control  provided  by  the  editing  software  allowed  experimenters  to  isolate 
certain  target  types  from  the  actions  of  any  computer-controlled  entity.  This  permitted  MTS 
instructors  and  SMEs  to  calculate  a  team’s  actual  mission  score  for  the  experimental  mission 
by  reviewing  their  observations  and  the  digitized  recordings  of  the  team’s  performance. 

For  example,  once  an  MTS  had  completed  their  assigned  mission  and  the  fifteen 
minutes  time  had  elapsed,  the  subjects  were  directed  to  move  to  their  planning  area  to  wait 
for  their  mission  debrief.  Given  that  it  normally  took  the  two  instructors  a  few  minutes  to 
compare  their  notes  and  score  the  mission,  subjects  could  take  a  brief  break  between 
missions.  Using  digital  recordings  of  the  mission  and  a  listing  of  the  mission's  primary  and 
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secondary  targets  (see  Table  4),  the  two  instructor  SMEs  were  able  to  jointly  score  the 
mission  based  on  the  number  and  type  of  targets  destroyed  by  the  subject  teams. 

As  discussed  earlier,  this  process  was  enhanced  by  the  fact  that  the  assigned  primary 
and  secondary  targets  could  only  be  destroyed  by  the  MTS.  An  important  aspect  of  these 
scoring  procedures  was  that  the  two  SMEs  compared  the  results  from  both  digital  recordings 
—  one  from  the  lead  aircraft  and  the  other  from  the  wing.  If  the  two  aircraft  were  widely 
separated  from  one  another  during  the  mission,  it  was  possible  for  one  recording  to  list  an 
assigned  target  as  destroyed  while  no  mention  was  made  of  it  on  the  second  recording.  By 
comparing  the  time  the  targets  were  destroyed,  as  they  appeared  on  the  digital  recording,  the 
SMEs  were  able  to  quickly  resolve  any  contradictions  and  calculate  the  MTS's  mission 
scores. 

Applying  the  target  values  from  a  previously  developed  scoring  table  (see  Table  16), 
instructors  were  able  to  quickly  calculate  an  overall  performance  score  for  the  two-dyad 
MTS.  As  can  be  seen  in  Table  16,  if  one  aircraft  within  the  MTS  survived,  but  was  damaged, 
while  the  other  team  was  undamaged,  the  MTS  received  50  points.  If  the  team  destroyed 
four  of  their  eight  assigned  targets,  they  received  80  points.  If  they  also  destroyed  three  of 
the  seven  secondary  targets,  they  gained  another  30  points.  Finally,  if  members  of  the  MTS 
also  destroyed  two  neutral  targets,  they  would  lose  80  points.  Based  on  these  point  totals, 
this  MTS's  overall  performance  score  would  be  80  points  (i.e.,  50  +  80  +  30  -  80).  These 
summed  inputs  provided  the  information  for  the  mission  debriefs  and  the  MTS  performance 
data  needed  to  test  the  hypotheses  described  in  this  study. 
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Chapter  5 
RESULTS 

Descriptive  statistics  and  intercorrelations  for  all  the  predictor  variables  used  to  test 
the  hypothesized  model  described  earlier  (see  Fig  2)  are  reported  in  Tables  17  and  18, 
respectively.  As  described  earlier,  a  2  (complexity)  x  3  (training  and  control)  x  3  (time) 
mixed  model  ANOVA  was  conducted  to  verify  that  time  was  not  a  significant  factor  in  the 
analysis.  Results  of  this  previous  analysis  (see  Table  5)  indicated  that  neither  mission  order 
nor  timing  significantly  influenced  MTS  performance. 

Repeated  measures  multiple  regression  (RMMR)  analyses  were  used  to  test  the 
relationships  presented  in  the  hypothesized  model.  Unlike  traditional  regression  techniques, 
repeated  measures  multiple  regression  allows  for  the  distribution  of  variance  among  within 
and  between  team  variables  (Cobb,  1997;  Goodwin,  1997;  Gully,  1994;  Heffner,  1996).  This 
adjusts  the  F  values  depending  on  whether  the  variable  occurs  within  or  between  subjects. 
These  adjustments  minimize  the  overestimation  of  between  subjects  effects  and  the 
underestimation  of  within  subjects  effects  normally  associated  with  repeated  measures 
regressions  (see  Cohen  &  Cohen,  1983;  Gully,  1994;  Heffner,  1996). 

In  this  study,  the  input  variables,  environmental  complexity  and  two  training 
conditions  (i.e.,  strategy  and  coordination),  were  between  team  variables.  MTS. performance 
(i.e.,  outcome)  and  the  three  process  variables,  i.e.  interpersonal,  action,  and  transition,  were 
within  team  variables.  Unless  otherwise  specified,  all  analyses  used  to  test  the  hypothesized 
model  occurred  at  the  MTS  level.  Therefore,  teams  in  this  analysis  refer  to  the  MTSs 
discussed  earlier,  i.e.  two  interdependent  dyads,  rather  than  single  dyads  comprised  of  two 
individuals.  RMMR  provides  a  powerful  tool  to  examine  the  unique  variance  contributed  by 
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the  variables  measured  in  this  study. 

Model  Tests. 


To  test  the  hypotheses  included  in  the  prescribed  model,  it  was  necessary  to  create  a 
stacked  data  set  containing  process  and  performance  data  from  all  appropriate  sources. 
Although  this  procedure  commonly  results  in  a  loss  of  independence  between  variables, 
Cohen  and  Cohen  (1983)  and  Gully  (1994)  recommend  adjusting  the  F  values  to  compensate. 
Using  RMMR,  these  adjusted  F  values  were  used  in  all  subsequent  tests  of  the  model  and  its 
variables.  Figure  2  has  been  reprinted  here  to  provide  an  easy  reference  while  reviewing  the 


results  of  the  analysis. 


Inherent  in  the  I-P-0  framework  is  the  assertion  that  the  effects  of  input  variables  on 
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outcome  are  mediated  by  process.  The  same  holds  true  for  the  hypotheses  proposed  in  this 
study.  To  test  for  mediation  effects,  some  method  for  operationalizing  mediation  must  be 
employed.  Based  on  earlier  work  by  Baron  and  Kenny  (1986),  Kenny,  Kashy,  and  Bolger 
(1997)  described  a  four-step  procedure  for  determining  if  mediation  is  present  within  a 
specified  framework.  This  procedure  was  used  in  this  study  to  test  for  all  mediation  effects 
present  in  the  hypothesized  model. 

According  to  Kenny  et  al.  (1997),  the  first  step  in  the  procedure  establishes  the  fact 
that  there  is  an  effect  to  be  mediated.  This  is  accomplished  by  regressing  the  outcome 
variable  upon  the  input  variables  to  determine  if  the  inputs  are  significantly  affecting 
outcomes.  If  there  is  no  direct  main  effect,  then  mediation  is  not  possible  since  there  isn't 
anything  to  mediate.  If  a  direct  main  effect  is  found,  then  one  must  establish  that  the 
independent  variable  predictor  is  significantly  related  to  the  proposed  mediator  (i.e..  Step  2). 
This  step  involves  treating  the  mediator  as  an  outcome  variable  in  the  analysis.  In  the  third 
step,  it  must  be  shown  that  the  mediator  affects  the  outcome  variable,  controlling  for  the 
input  variables.  Again,  the  outcome  variable  is  regressed  upon  the  mediator  variables  to 
determine  if  any  significant  relationship  exists. 

Using  the  I-P-0  framework,  the  final  step  described  by  Kenny  et  al.  (1997) 
determines  if  process  completely  mediates  the  effects  of  the  input  variables.  Complete 
mediation  is  indicated  when  the  regression  of  the  outcome  variable  on  the  input  variables, 
controlling  for  the  mediator's  effects,  is  reduced  to  non-significance  or  nearly  zero  (Kenny  et 
al.).  According  to  the  authors,  if  the  first  three  steps  are  met,  but  not  the  fourth,  then  partial 
mediation  exists.  In  this  case,  the  relationship  between  the  predictor  variable  and  the 
outcome  variable  is  best  described  as  indirect  and  being  conveyed  by  the  moderator.  If  this 
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were  the  case,  training  could  be  seen  as  having  an  enabling  effect  upon  action  process 
behaviors  that  enhances  overall  MTS  performance.  The  hypothesized  effects  of  the  two 
training  conditions  in  this  study  provide  an  example  of  complete  mediation. 

Following  the  four-step  procedure  described  by  Kenny  et  al.  (1997),  a  series  of 
ANOVAs  was  completed  to  identify  any  evidence  of  mediation  within  the  proposed  model. 
As  summarized  by  Table  19,  a2x3x3  mixed  model  ANOVA  was  used  to  determine  if  any 
significant  relationship  existed  between  the  input  variables  (i.e.,  environmental  complexity, 
strategy  training,  and  coordination  training)  and  the  outcome  variable  (i.e.,  MTS 
performance).  Since  MTS  performance  was  measured  in  three  separate  observations,  time 
was  included  in  the  analysis  to  examine  any  possible  interactions  with  the  input  variables. 
Based  on  this  analysis,  no  significant  relationship  existed  between  the  input  variables  and  the 
outcome  measures  used  in  this  study  (Table  19).  Therefore,  the  training  and  complexity 
manipulations  described  earlier  had  no  significant  direct  impact  on  MTS  performance.  The 
hypothesized  mediation  was  not  possible  since  there  was  no  direct  relationship  to  be 
mediated  between  the  input  and  outcome  variables. 

Although  there  was  no  basis  for  expecting  a  mediation  effect  between  the  input  and 
outcome  variables,  the  relationships  hypothesized  in  the  proposed  model  were  further 
examined  for  evidence  of  significant  direct  effects  among  the  variables.  Tables  20-22 
summarize  the  results  of  the  ANOVAs  examining  the  potential  relationships  between  the 
input  variables  and  the  three  process  measures  (i.e.,  interpersonal,  action,  and  transition). 

This  analysis  indicated  that  the  input  manipulations  also  failed  to  have  any  significant  direct 
effect  upon  any  of  the  process  variables  represented  in  the  hypothesized  model.  The  only 
significant  relationships  found  in  this  analysis  involved  a  two-way  interaction  between 
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complexity  and  time  (F(2, 29)  =  6.79,  p<.01)  and  a  three-way  interaction  between 
time*complexity*training  (F(4,  58)  =  2.77,  p<.05)  on  interpersonal  process  behaviors. 

Based  on  these  results,  it  appeared  that  complexity  adversely  affected  interpersonal 
process  behaviors  over  time  (see  Table  20).  More  specifically,  subjects  in  the  low 
complexity  condition  seemed  to  improve  their  interpersonal  process  behaviors  over  time 
while  those  in  the  higher  condition  seemed  to  fall  apart  over  time  (see  Tables  23-24). 
Figures  3-5  illustrate  the  results  summarized  by  Tables  23-24  for  each  of  the  training 
conditions. 

Figure  3:  Coordination  Training 
Coordination  Training 


TIME 
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Figure  4:  Strategy  Training 


Strategy  Training 


TIME 

Figure  5:  Control  Condition 


Control 


TIME 

These  results  were  similar  to  those  found  in  previous  studies  reviewed  by  Pelled 
(1996)  and  seemed  to  indicate  that  the  MTS  began  to  experience  more  inteipersonal 
problems  over  time  under  conditions  of  higher  levels  of  complexity.  More  complex 
environments  could  have  increased  the  amount  of  negative  conflict  present  within  the  team, 
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which  adversely  affected  their  interpersonal  behaviors  (see  Pelled,  1996). 

The  reduction  in  the  quality  of  the  teams'  interpersonal  from  the  first  to  the  second 
mission  noted  in  the  low  complexity,  coordination  training  condition  (see  Figure  3)  appears 
to  be  an  aberration,  when  compared  to  the  other  two  conditions.  Previous  research  by 
Gersick  (1988, 1989)  found  that  groups  tend  to  pace  themselves  over  time.  According  to  her 
findings,  groups  seemed  motivated  by  their  awareness  of  time  and  deadlines  and  deliberately 
shifted  their  attention  at  their  temporal  midpoint.  This  could  apply  to  teams  in  the 
coordination  training,  since  the  ROTC  teams  comprising  this  condition  may  have  been  more 
concerned  about  their  combined  performance  score  for  all  three  missions,  due  to  the 
recognition  awards  used  for  recruitment,  than  the  undergraduate  psychology  students  in  the 
other  conditions  or  later  ROTC  teams.  This  could  have  contributed  to  any  perceptions  they 
might  have  had  that  the  three  missions  were  subtasks  of  a  greater  whole.  Additional  data  and 
research  is  needed  to  discern  if  this  tendency  could  have  contributed  to  the  pattern  of  results 
noted  here. 

Hypothesized  Model.  The  initial  hypotheses,  i.e.  H1-H3,  asserted  that  effective  team 
processes  would  be  associated  with  higher  levels  of  MTS  performance.  Based  on  Marks  et 
al.’s  (in  preparation)  description,  it  was  believed  that  how  well  teams  engaged  in  pre-mission 
planning,  action  process  behaviors,  and  interpersonal  relations  would  have  direct  effects  on 
their  performance.  As  seen  in  the  proposed  model,  this  meant  that  transition  processes 
should  influence  overall  performance  in  addition  to  impacting  interpersonal  and  action 
process  behaviors.  Following  the  sequence  of  the  variables  presented  in  the  hypothesized 
model,  interpersonal  and  action  process  were  entered  into  the  regression  equation  before 
transition  process  and  complexity.  Appendix  K  presents  a  complete  summation  of  the 
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RMMR  results  and  significance  testing  for  the  hypothesized  model. 

Based  on  the  results  from  the  RMMR,  transition  and  interpersonal  process  behaviors 
had  no  direct  main  effect  on  MTS  performance  (beta  =  .058,  F(l,  68)  =  .57,  p  <  .05  and  - 
.117,  F(l,  68)  =1.93,  p  <  .05,  respectively)  (see  Table  25).  Action  process  behaviors,  on  the 
other  hand,  significantly  (beta  =  .393,  F(l,  68)  =  20.44,  p  <  .01)  impacted  how  well  teams 
performed  in  the  mission  simulations  and  seemed  to  carry  most  of  the  variance  due  to  team 
process.  Thus,  hypothesis  2  was  supported  by  the  analysis,  while  hypotheses  1  and  3  were 
not. 

It  was  hypothesized  that  complexity  would  negatively  impact  team  performance  both 
directly  and  through  team  processes  (i.e.,  H6).  The  data  analyzed  in  this  study  failed  to 
provide  support  for  this  hypothesis.  Complexity  was  not  significantly  related  to  MTS 
performance  (beta  =  -.146,  F(l,  34)  =  1.41,  p>.05).  The  hypothesized  interactions  between 
complexity  and  strategy  training  (i.e.,  HI  1)  and  between  complexity  and  coordination 
training  (i.e.,  HI 2)  also  failed  to  significantly  influence  overall  team  performance  (beta  =  - 
.062,  F(l,  30)  =  0,  p>.05;  beta  =  -.464,  F(l,30)  =  .95,  p>.05,  respectively)  (see  Table  25). 

The  next  phase  of  analysis  examined  the  variable  hypothesized  to  influence  MTS 
interpersonal  behaviors.  As  expected,  transition  process  behaviors  directly  impacted 
interpersonal  processes  (beta  =  .147,  F(l,  70)  =  16.08,  p  <  .01)  (see  Table  26).  While 
hierarchical  regression  failed  to  note  the  significance  of  this  relationship,  RMMR  analysis 
corrected  the  underestimation  of  within  team  effects  and  provided  support  for  the  hypothesis 
(i.e.,  H5). 

Following  the  proposed  model,  variables  were  tested  for  their  influence  on  MTS 
action  processes.  It  was  hypothesized  that  the  quality  of  teams’  planning  strategies  (i.e., 
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transition  process  behaviors)  would  enhance  their  action  process  behaviors  (i.e.,  H4).  It  was 
also  believed  that  teams  receiving  coordination  training  would  exhibit  more  effective  action 
process  behaviors  (i.e.,  H8)  than  those  who  did  not  receive  such  training.  Finally,  it  was 
hypothesized  that  environmental  complexity  would  reduce  the  benefits  of  coordination 
training  on  action  process  behaviors  (i.e.,  H10). 

RMMR  analysis  indicated  that  only  the  first  of  these  hypotheses  was  supported  by  the 
data  (see  Table  27).  Although  transition  process  behaviors  were  found  to  significantly  affect 
action  process  (beta  =  .26,  F(l,  71)  =  14.62,  p  <  .01),  RMMR  analysis  showed  that 
coordination  training  did  not  significantly  influence  action  process  behaviors  (beta  =  .228, 
F(l,34)  =  3.02,  p>.05).  The  interaction  between  complexity  and  coordination  training  was 
also  found  to  be  an  insignificant  contributor  to  action  process  behaviors  (beta  =  -.542,  F(l, 

32)  =  1.18  p>.05)  (see  Table  27). 

Finally,  analyses  were  conducted  to  determine  if  team  training  significantly 
contributed  to  team’s  transition  process  behaviors.  It  was  hypothesized  that  strategy  training, 
which  focused  on  planning  and  mission  analysis  behaviors,  would  significantly  enhance 
teams'  transition  process  behaviors  (i.e.,  H7).  RMMR  analysis  indicated  (see  Table  28)  that 
there  was  no  support  for  this  hypothesis  (beta  =  -.129,  F(l,  34)  =  .78,  p>.05).  In  fact,  closer 
examination  of  the  analysis  indicated  that  strategy  training  might  actually  negatively 
influence  teams’  transition  process.  It  was  also  hypothesized  that  the  interaction  of 
complexity  and  strategy  training  would  significantly  influence  transition  process  behaviors 
(i.e.,  H9).  This  hypothesis  was  also  unsupported  by  the  RMMR  analysis  (beta  =  .701,  F(l, 

32)  =  1.78,  p>.05). 

Following  the  recommendations  of  Cohen  and  Cohen  (1983),  additional  regression 
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analyses  were  completed  to  determine  if  any  relationships  not  represented  in  the 
hypothesized  model  were  significant.  This  was  accomplished  by  first  regressing  the 
dependent  variable  onto  those  independent  variables  found  as  significant  in  the  previous 
RMMR  analyses.  Once  these  variables  were  entered,  stepwise  regression  was  employed  to 
identify  any  additional  significant  relationships  between  the  dependent  variable  and  the 
remaining  variables.  According  to  Cohen  and  Cohen  (1983),  stepwise  regressions  are 
conducted  separately  from  hypothesis  testing  and  define  "...a posteriori  order  based  solely  on 
the  relative  uniqueness  of  the  variables  in  the  sample. . . "  (p.  123).  In  this  way,  the  number  of 
insignificant  relationships  can  be  reduced  to  increase  the  chance  of  identifying  previously 
unrecognized  significance  levels  existing  between  variables  in  the  hypothesized  model.  This 
analysis  indicated  that  no  other  significant  relationships  existed  in  the  model. 

Trimmed  Model.  Previous  research  has  found  that  deleting  nonsignificant  paths  from 
a  model  can  enhance  further  tests  of  relationships  that  were  previously  identified  as  being 
close  to  significant  levels  (e.g.,  Mathieu,  1988).  Based  on  the  previous  analysis,  the 
hypothesized  model  was  reduced  as  seen  in  Figure  6. 

Figure  6:  Trimmed  Model 
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While  interpersonal  and  transition  process  behaviors  had  been  shown  earlier  as  not 
having  any  significant  effect  on  MTS  performance,  they  were  retained  in  the  trimmed  model 
to  see  if  they  gained  significance  after  eliminating  complexity  from  the  analysis.  This  was 
possible  since  the  previous  analyses  had  shown  that  subjects  had  failed  to  perceive  any 
significant  difference  between  the  two  complexity  conditions.  This  trimmed  model  was 
examined  using  the  same  procedures  used  to  test  the  hypotheses  in  the  original  model. 

As  in  the  original  analysis,  the  trimmed  model  was  first  tested  to  discover  if 
coordination  training's  effects  on  MTS  performance  were  being  mediated  by  action  process 
behaviors,  as  indicated  in  Figure  6.  Following  Kenny  et  al.'s  (1997)  recommendations,  MTS 
performance  was  regressed  onto  coordination  training,  controlling  for  the  process  variables. 
As  found  earlier,  the  results  were  not  significant  (beta  =  .125,  F(l,34)  =  1.12,  p>.05), 
Therefore,  there  was  no  evidence  supporting  mediation  of  coordination  training  by  action 
process  behaviors. 

The  remaining  hypotheses  were  tested  using  RMMR  and  the  results  were  nearly 
identical  to  the  original  analysis.  Of  the  three  process  variables,  only  action  process 
behaviors  (beta  =  .309,  F(l,  68)  =  19.21,  p<.01)  directly  influenced  MTS  performance  (see 
Table  29).  Transition  process  behaviors  significantly  contributed  to  both  interpersonal  (beta 
=  .147,  F(l,  70)  =  16.08,  p<.01)  and  action  (beta  =  .26,  F(l,  71)  =  14.62,  p<.01)  process 
behaviors.  Once  again,  coordination  training  failed  to  significantly  contribute  to  action 
process  behaviors  (beta  =  .228,  F(l,  34)  =  3.02;  p>.05).  Subsequent  analysis  also  failed  to 
identify  any  additional  potentially  significant  relationships  in  the  trimmed  model. 

Reviewing  the  results  for  the  original  and  trimmed  model,  one  additional  possibility 
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became  apparent  regarding  the  lack  of  effects  found  for  transition  process  behaviors  on  MTS 
performance.  If  transition  process  behaviors  occur  before  action  process  behaviors,  as 
represented  in  the  model  and  described  by  Marks  et  al.  (in  preparation),  their  direct  effects  on 
team  performance  (i.e.,  HI)  could  have  been  mediated  by  action  process  behaviors.  Once 
again,  the  procedures  recommended  by  Kenny  et  al.  (1997)  were  used  to  test  this  possibility. 

First,  MTS  performance  was  regressed  onto  transition  process  behaviors,  controlling 
for  the  effects  of  action  process  behaviors.  RMMR  analysis  indicated  that  transition  process 
behaviors  significantly  contributed  to  MTS  performance  (beta  =  . 171,  F(  1,71)  =  4.32,  p<.05) 
when  the  effects  of  action  process  behaviors  were  controlled.  The  previous  analysis  testing 
the  hypotheses  represented  in  the  model  established  the  fact  that  transition  process  behaviors 
direct  impacted  action  process  behaviors  (step  2  of  Kenny  et  al.  (1997))  and  action  process 
behaviors  impacted  MTS  performance  (step  3).  The  final  step  of  Kenny  et  al.'s  procedures 
was  completed  to  determine  the  level  of  mediation  present  in  the  relationship.  According  to 
this  analysis,  the  effects  of  transition  process  behaviors  on  team  performance  were  reduced  to 
insignificance  (beta  =  .058,  F(l,  69)  =  .61,  p>.05)  when  action  process  behaviors  were 
entered  into  the  regression.  According  to  Kenny  et  al.,  this  indicated  that  the  effects  of 
transition  process  behaviors  on  MTS  performance  were  completed  mediated  by  action 
process  behaviors. 

Employing  Continuous  Scores  for  Training  Manipulations.  To  this  point  in  the 
analysis,  dummy  codes  had  been  used  to  identify  the  appropriate  training  condition  (i.e., 
coordination,  strategy,  or  control)  for  the  MTS.  These  dichotomous  measures  may  be  too 
limited  to  adequately  capture  the  underlying  variance  in  the  model  due  to  training  conditions. 
Alternative  measures  for  the  training  conditions  were  available  in  the  data.  As  discussed 
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earlier,  subjects'  knowledge  of  the  critical  KSAs  associated  with  team  strategy  and 
coordination  was  assessed  by  three  questions  used  as  manipulation  checks  for  the  training 
manipulations. 

As  earlier  analysis  indicated  (see  Table  9),  these  knowledge  measures  evidenced  the 
expected  mean  patterns  corresponding  to  the  training  manipulation,  thus  demonstrating  some 
construct  validity  for  the  intended  knowledge  states  resulting  from  the  training  condition. 
These  measures  also  reflected  the  fact  that  the  MTSs  may  differ  in  how  much  understanding 
and  mastery  they  have  in  each  of  the  areas  addressed  by  the  training  conditions,  regardless  of 
the  specific  condition  they  were  placed  within.  Since  these  measures  may  better  capture  the 
underlying  constructs  described  in  the  hypothesized  model,  the  analysis  was  reaccomplished 
substituting  the  manipulation  check  scale  scores  for  the  dummy  coded  training 
manipulations. 

Again  following  Kenny  et  al.'s  (1997)  recommendations,  MTS  performance  was 
regressed  onto  transition  process  behaviors  and  the  three  input  variables  in  separate 
regressions.  As  found  previously,  RMMR  analysis  indicated  that  transition  process 
behaviors  significantly  contributed  to  MTS  performance  (beta  =  .171,  F(l,  71)  =  4.32,  p<.05) 
when  the  effects  of  action  process  behaviors  were  controlled.  The  second  RMMR  analysis 
indicated  that  only  coordination  training  significantly  impacted  MTS  performance  (beta  = 
.267,  F(l,  32)  =  3.37,  p<.05). 

These  results  provided  some  initial  support  for  the  results  found  previously  and  the 
model's  assertions  that  the  effects  of  coordination  training  on  team  performance  were 
mediated  by  team  process.  Testing  the  hypotheses  represented  in  the  model  fulfilled  the 
requirements  of  the  next  two  steps  of  Kenny  et  al.'s  procedures  -  verifying  that  the  mediator 
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was  significantly  related  to  outcomes  and  the  input  variables.  Once  these  relationships  were 
tested  and  established,  the  final  step  in  the  procedure  was  accomplished  to  see  if  complete 
mediation  existed.  Appendix  L  presents  a  complete  summation  of  the  RMMR’s  results  and 
significance  testing  for  the  hypothesized  model  using  continuous  scale  scores  as  measures  of 
the  MTS'  knowledge  states  derived  from  the  training  conditions. 

As  described  earlier,  the  first  three  hypotheses  (i.e.  H1-H3)  asserted  that  effective 
team  process  behaviors  would  be  associated  with  higher  levels  of  MTS  performance. 
Following  the  sequence  of  the  variables  presented  in  the  hypothesized  model  and  used  in  the 
original  analysis,  interpersonal  and  action  process  were  entered  into  the  regression  equation 
before  transition  process  and  complexity.  As  in  the  original  analysis,  only  H2  was  supported 
by  the  RMMR.  Action  process  behaviors  significantly  impacted  MTS  performance  (beta  = 
.393,  F(l,  68)  =  20.44,  p<.01),  while  transition  and  interpersonal  process  behaviors  had  no 
direct  main  effect  on  performance  (beta  =  .058,  F(l,  68)  =  .57,  p>.05,  and  -.117,  F(l,  68)  = 
1.93,  p>.05,  respectively;  see  Table  30).  Thus,  hypotheses  1  and  3  were  not  supported  by  the 
RMMR  analysis. 

It  was  hypothesized  that  complexity  would  negatively  impact  team  performance  both 
directly  and  through  team  processes  (i.e.,  H6).  The  results  of  the  RMMR  indicated  that 
complexity  was  not  significantly  related  to  MTS  performance  (beta  =  -.146,  F(l,  34)  =  1.14, 
p>.05).  As  in  the  original  analysis,  the  hypothesized  interactions  between  complexity  and 
strategy  training  (i.e.,  HI  1)  and  between  complexity  and  coordination  training  (i.e.,  HI 2)  also 
failed  to  significantly  influence  team  performance  (beta  =  .026,  F(l,  30)  =  0  and  beta  =  -.169, 
F(l,  30)  =  .07,  p>.05,  respectively)  (see  Table  30). 

As  found  earlier,  transition  process  behaviors  directly  impacted  interpersonal  (beta  = 
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.147,  F(l,  70)  =  16.08,  p  <  .01)  and  action  (beta  =  .227,  F(l,  71)  =  10.07,  p  <  .01)  (see  Tables 
31-32)  process  behaviors.  Thus,  H4  and  H5  were  again  supported  by  the  RMMR  analysis. 

As  discussed  earlier,  it  was  also  believed  that  teams  receiving  coordination  training  would 
exhibit  more  effective  action  process  behaviors  (i.e.,  H8),  while  environmental  complexity 
would  reduce  the  benefits  of  this  training  for  action  process  behaviors  (i.e.,  H10).  Although 
H10  was  not  supported  (beta  =  .166,  F(l,  32)  =  0) ,  the  RMMR  analysis  indicated  that 
coordination  training  had  a  significant  direct  effect  on  action  process  behaviors  (beta  =  .279, 
F(l,  34)  =  4.59,  p<.05)  (see  Table  32).  This  effect  was  not  supported  in  the  original  analysis 
using  dummy  codes  to  represent  training  conditions. 

Finally,  analyses  were  conducted  to  determine  if  strategy  training  significantly 
contributed  to  teams’  transition  process  behaviors  (i.e.,  H7).  RMMR  analysis  indicated  (see 
Table  33)  that  there  was  no  support  for  this  hypothesis  (beta  =  -.129,  F(l,  34)  =  1 .01,  p>.05). 
As  discussed  earlier,  it  was  also  hypothesized  that  the  interaction  of  complexity  and  strategy 
training  would  significantly  influence  transition  process  behaviors  (i.e.,  H9).  This  hypothesis 
was  also  unsupported  by  the  RMMR  analysis  (beta  =  .078,  F(1 ,  32)  =  .13,  p>.05). 

Again,  following  Cohen  and  Cohen's  (1983)  recommendations  and  the  same 
procedures  used  in  the  original  analysis,  additional  regression  analyses  were  completed  to 
determine  if  any  relationships  not  represented  in  the  hypothesized  model  were  significant. 
This  analysis  indicated  that  coordination  training  was  potentially  influencing  interpersonal 
and  transition  process  behaviors.  RMMR  analyses  indicated  that  there  was  no  significant 
relationship  between  coordination  training  and  transition  process  behaviors  (beta  =  .253,  F(l, 
34)  =  2.09,  p>.05).  However,  these  analyses  did  indicate  that  coordination  training  directly 
affected  interpersonal  process  behaviors  (beta  =  .324,  F(l,  34)  =  4.25;  p<.05)  in  addition  to 
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its  impact  on  action  process  behaviors. 

These  final  tests  of  the  hypothesized  model  also  completed  steps  two  and  three  of 
Kenny  et  al.'s  (1997)  check  for  mediation.  As  discussed  earlier  in  this  analysis,  not  only  was 
coordination  training  significantly  related  to  MTS  performance,  controlling  for  the  process 
mediators,  but  it  was  also  directly  related  to  two  process  variables  (i.e.,  interpersonal  and 
action  process  behaviors).  However,  step  3  of  these  procedures  required  the  mediators  to 
significantly  affect  the  outcome  variable  (i.e.,  MTS  Performance)  (Kenny  et  al.,  1997).  As 
discussed  earlier,  only  action  process  significantly  contributed  to  team  performance.  RMMR 
analysis  indicated  that  when  action  process  behaviors  were  entered  into  the  regression, 
coordination  training  completely  lost  its  previous  significance  (from  beta  =  .236,  F(l,  32)  = 
4.27,  p<.05;  to  beta  =  .108,  F(l,  32)  =  .83,  p>.05).  Therefore,  the  effects  of  coordination 
training  on  team  performance  were  completely  mediated  by  action  process  behaviors. 

As  expected,  since  none  of  the  original  values  were  altered,  the  results  from  the 
RMMR  analysis  were  identical  to  those  found  previously  regarding  the  tests  for  mediation 
between  action  and  transition  process  behaviors.  Action  process  behaviors  completely 
mediated  the  effects  of  transition  process  onto  MTS  performance.  Figure  7  summarizes  the 
findings  from  this  analysis. 
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Figure  7:  Analysis  Summaiy 


Exploratory  Analysis. 


Although  some  basic  exploratory  analysis  had  already  been  accomplished  in  testing 
the  hypotheses  and  mediations  represented  in  the  proposed  model,  several  questions 
remained.  One  of  these  focused  on  the  nature  of  the  measure  used  to  indicate  the  quality  of 
the  MTS'  transition  process  behaviors  in  the  original  hypothesis  testing.  As  described  earlier, 
this  measure  was  aggregated  from  SME  interview  ratings,  in-basket  exercise  scores,  and  the 
appropriateness  of  the  teams'  weapon  selection.  Since  the  interview  components  of  this 
composite  score  represented  the  team  leader’s  summation  of  the  entire  planning  session,  it 
was  thought  that  they  might  provide  an  alternative  to  the  existing  composite  measure.  To  test 
whether  this  was  true,  the  original  hypotheses  were  retested  using  the  mean  of  the  SME 
ratings  from  the  mission  commander  interviews  as  the  transition  process  behaviors  measure. 

Following  the  same  procedures  used  to  originally  test  the  proposed  model,  each  of  the 
proposed  hypotheses  was  reexamined  using  RMMR  analysis.  The  results  of  these  analyses 
are  summarized  in  Appendix  M.  As  can  be  seen  by  comparing  the  original  tests  of  the 
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hypothesized  model  with  those  using  the  aggregated  interview  ratings  as  the  transition 
process  measure  (i.e.,  Appendix  K  with  Appendix  M),  these  results  showed  some  differences 
from  the  original  analysis.  While  the  retest  largely  duplicated  the  original  results,  it  failed  to 
identify  any  mediation  effects  or  additional  areas  of  significance  (see  Tables  34  -  37). 

Based  on  these  analyses,  neither  the  training  or  complexity  manipulations  had  any 
significant  effect  on  MTS  performance  or  team  process  (see  Tables  34  -37).  Action  process 
behaviors  remained  the  only  significant  contributor  to  team  performance  (beta  =  .404,  F(l, 
68)  =  21.04,  p<.01)  (see  Table  34).  Unlike  the  previous  two  tests  of  the  model,  transition 
process  behaviors  did  not  have  any  significant  direct  effect  on  team  performance,  when 
action  process  behaviors  were  controlled  in  the  analysis  (beta  =  .149,  F(l,  70)  =  3.23,  p>.05). 
Therefore,  there  was  no  evidence  of  any  mediation  in  the  proposed  model  when  interview 
scores  alone  were  used  as  measures  of  transition  process.  Transition  process  behaviors  were 
found  to  directly  impact  both  interpersonal  (beta  =  .149,  F(l,  68)  =  16.08,  p<.01)  and  action 
(beta  =  .26,  F(l,  71)  =  14.35,  p<.01)  process  behaviors  (see  Tables  35  -  36). 
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Chapter  6 
DISCUSSION 

The  purpose  of  this  study  was  to  expand  on  previous  research  examining  how 
manipulating  the  level  of  environmental  complexity  influence  MTS  process  behaviors  and 
performance  outcomes.  It  also  tested  critical  aspects  of  a  new  team  process  framework  being 
developed  by  Marks  et  al.  (in  preparation)  and  offers  insights  into  potential  modifications  of 
the  existing  model.  Specifically  it  focused  on  assessing  how  subject  teams  performed  in  a 
dynamic  environment  that  not  only  required  cooperative  behavior  between  individuals  within 
a  single  team,  but  also  collective  actions  between  separate  teams  to  achieve  assigned 
performance  goals.  The  results  from  this  study  contribute  to  the  body  of  work  supporting 
and  guiding  the  emerging  theory  of  MTSs.  It  also  expands  current  research  employing  low- 
fidelity  personal  computer  (PC)  flight  simulations  to  study  team  processes  and  performance 
(e.g.,  Goodwin,  1997;  Mathieu  &  Goodwin,  1997;  Stout,  Cannon-Bowers,  Salas,  &  Morgan, 
1990;  Kozlowski  &  Gully,  1996)  into  the  multi-team  environment. 

The  results  of  this  study,  as  summarized  in  Table  37,  partially  support  key  aspects  of 
the  hypothesized  framework  by  shedding  light  on  the  nature  of  the  relationships  between 
team  process  variables.  Key  among  these  findings  was  support  for  sequencing  the  process 
variables  in  a  way  that  recognized  the  central  role  played  by  transition  process  behaviors. 
Previous  research  has  shown  that  teams'  behaviors  change  over  time  as  they  make  decision 
and  engage  in  problem-solving  (see  Gersick,  1988).  Rather  than  emphasizing  problem¬ 
solving  in  isolation,  the  model  examined  in  this  study  places  these  behaviors  in  conjunction 
with  the  actions  required  to  execute  decisions  both  within  and  between  interdependent  teams. 
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Although  previous  research  has  consistently  illustrated  a  lack  of  a  strong  relationship 
between  interpersonal  process  behaviors  and  team  performance  (see  Cannon-Bowers  et  ah, 
1995;  Driskell  &  Salas,  1992),  findings  have  been  inconsistent  in  identifying  what  factors 
link  these  behaviors  to  more  action  oriented  activities.  The  analyses  in  this  study  indicate 
that  transition  behaviors,  as  described  by  Marks  et  al.  (in  preparation),  may  provide  such  a 
link  between  interpersonal  and  action  process  behaviors. 

Based  on  the  results  of  this  study,  transition  process  behaviors  could  be  viewed  as 
important  enabling  activities  that  enhance  other  important  process  behaviors  within  the  team 
setting.  By  understanding  how  their  teams’  actions  are  impacted  by  and  influence  the  actions 
of  other  teams,  teams  within  an  MTS  appear  better  able  to  carry  out  their  plans  of  action  to 
achieve  task  objectives  (see  Marks  et  al.,  in  preparation).  Expanding  on  this  theoretical 
foundation,  the  results  from  this  study  indicate  that  the  primary  role  of  transition  process 
behaviors  does  not  lie  in  its  direct  effect  on  team  performance,  which  was  found  to  be 
completely  mediated  by  action  process  behaviors.  Instead,  transition  process  behaviors  seem 
to  provide  critical  links  between  action  and  interpersonal  processes  within  MTSs  and  the 
teams  comprising  them.  This  finding  is  unique  and  an  important  contribution  to  the  literature 
examining  team  process  and  performance. 

As  anticipated,  the  effects  of  both  coordination  training  and  transition  process 
behaviors  on  MTS  performance  were  mediated  by  action  process  behaviors.  Being  well 
trained  or  having  an  accurate  plan  of  action  prior  to  a  mission  could  only  directly  impact 
performance  if  the  MTS  was  able  to  employ  those  skills  or  successfully  implement  the  plans 
during  the  mission.  As  emphasized  by  previous  research  (e.g.,  Mathieu  et  al.,  1996;  Swezey 
&  Salas,  1992),  the  actions  teams  take  greatly  influence  their  subsequent  actions  and  how 
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well  they  perform  on  interdependent  tasks.  Based  on  the  results  of  this  study,  previous 
planning  and  training  can  enhance  the  effectiveness  of  a  MTS  actions  by  increasing  their 
understanding  of  the  task  environment  and  each  team's  responsibilities  within  it.  These 
results  also  indicate  that  measures  reflecting  the  distributed  and  collective  knowledge  and 
expertise  existing  within  the  MTS  are  a  great  deal  more  effective  than  merely  noting  their 
training  condition  when  analyzing  the  outcomes  of  training  interventions  (c.g.,  Hollenbeck  et 
al.,  1995;  Liang,  Moreland,  &  Argote,  1995;  Moreland,  Argote,  &  Krishnan,  1997;  Weick  & 
Roberts,  1993).  Future  research  is  required  to  discern  just  how  closely  transition  and  action 
process  behaviors  must  be  aligned  (or  divergent)  to  enhance  overall  MTS  performance. 

Team-based  research  has  frequently  relied  on  subject  self-reporting  for  collecting  data 
to  test  various  hypotheses.  Although  SME  observations  and  ratings  were  an  important  part 
of  the  data  analyzed  in  this  study,  no  self-reported  data  was  employed  to  test  the  proposed 
hypotheses.  This  is  a  significant  departure  from  previous  research  and  reinforces  the 
assertion  that  multiple  sources  of  information,  i.e.  objective  and  subjective,  must  be  used  to 
adequately  capture  team  processes  and  performance. 

Most  surprising  and  somewhat  perplexing  is  the  fact  that  the  results  of  this  study 
failed  to  show  any  significant  effects  resulting  from  the  manipulation  of  environmental 
complexity.  As  discussed  earlier  in  this  paper,  Pelled  (1986)  and  others  have  argued  that 
negative  conflict  increases  in  more  demanding,  complex  work  environments.  In  fact,  the 
interactions  between  time*complexity*training  discussed  earlier  (see  Tables  23-24)  on 
interpersonal  process  seemed  to  indicate  that  complexity  was  impacting  team  process 
behaviors.  As  discussed  earlier,  the  failure  of  the  manipulations  could  indicate  that  greater 
variance  was  needed  between  the  two  conditions  before  subjects  could  perceive  any 


93 


significant  differences.  The  addition  of  only  eight  distracters,  i.e.  a  17%  increase  over  the 
low  complexity  condition  (see  Table  4),  may  not  have  been  sufficient  to  alter  the  subjects' 
perceptions  of  the  task  environment's  complexity  from  the  perceptions  they  formed  during 
training  and  the  practice  mission. 

Based  on  informal  feedback  received  during  training  from  participants  and  SME 
instructors,  most  participants  found  the  basic  training  and  practice  mission  (see  Table  10) 
extremely  challenging.  Being  placed  in  a  single  complexity  condition  could  have  limited  the 
variance  normally  associated  with  manipulations  of  this  type.  In  a  sense,  there  was  no 
opportunity  to  compare  a  single  MTS'  performance  and  process  behaviors  in  both  low  and 
high  complexity  conditions.  An  alternative  assessment  method  could  have  been  used  to  rate 
the  complexity  of  the  missions.  SMEs  could  have  been  used  to  rate  the  recordings  of  each 
mission  in  terms  of  their  complexity.  These  ratings  might  provide  a  more  sensitive  measure 
of  complexity  than  the  subjects'  perceptions  of  the  mission  after  completing  it.  Such  an 
analysis  is  beyond  the  scope  of  this  paper,  since  it  would  require  additional  data  collection. 

One  factor  that  could  have  impacted  the  results  found  in  this  study  concern  how 
subjective  assessments  of  transition  process  behaviors  were  conducted.  While  the  objective 
tests  used  in  this  study  provided  insights  into  teams’  reasoning  and  planning  strategies, 
subjective  SME  ratings  were  important  to  gaining  a  full  appreciation  of  the  effectiveness  of 
their  overall  transition  processes.  One  limitation  to  these  ratings  was  that  they  were  based  on 
an  interview  with  one  member  of  the  team  in  isolation  from  other  team  members  and  the 
MTS’  actions  during  the  experimental  simulation.  Without  having  observed  the  MTS’ 
deliberations  during  their  actual  planning  session,  it  could  have  been  difficult  to  accurately 
determine  if  the  results  of  the  interview  accurately  reflected  the  teams’  analysis  and  planning 
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or  that  of  a  single  individual.  This  could  have  undermined  the  effectiveness  of  the  SMEs’ 
efforts  to  assess  important  transition  behaviors  and  the  possible  impact  of  both  training 
conditions. 

At  the  start  of  each  mission  planning  session,  team  leaders  were  also  given  a  copy  of 
the  questions  to  be  used  during  the  interview.  It  was  thought  that  this  was  needed  to  expedite 
the  process  and  ensure  mission  commanders  remained  focused  and  on  schedule  during  the 
interview.  However,  a  closer  look  at  the  questions  used  in  the  interview  (see  Appendix  I) 
reveals  that  they  can  easily  serve  as  an  advanced  organizer  for  teams  to  use  in  their  planning 
sessions.  Although  not  specific  about  what  actions  they  need  to  take,  the  questions  do 
provide  insights  into  what  should  be  considered  in  strategic  planning.  Since  every  team 
received  these  questions  prior  to  each  mission  planning  session,  it  could  have  caused  the 
training  conditions  to  become  increasingly  similar  over  time  and  made  it  more  difficult  to 
isolate  the  effects  of  one  training  condition  from  another. 

Finally,  each  of  the  training  conditions  used  equivalent  presentations,  in  terms  of 
length  and  depth,  to  instruct  the  subjects  in  this  study.  However,  such  a  manipulation  may 
have  overlooked  the  fact  that  subjects  could  have  required  greater  depth  in  one  area  than  the 
other  before  they  could  employ  these  skills.  Given  the  normal  life  experiences  of  most 
undergraduates,  it  is  probably  safe  to  say  that  they  could  have  had  a  higher  level  of 
competency  in  employing  coordination  skills  in  a  group  setting  than  those  associated  with 
strategy  formulation  and  planning.  Although  the  manipulation  check  provided  a  basic 
measure  of  the  knowledge  of  these  concepts  following  training,  they  might  not  be  sensitive 
enough  measures  to  identify  varying  levels  of  competency  within  the  teams. 

Training  on  coordination  skills  could  have  been  easier  for  most  subjects  to  assimilate 
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and  relate  to  performance  outcomes  due  to  their  past  experiences  (c.g.,  Lord  &  Maher,  1991; 
Moreland,  Argote,  &  Krishnan,  1996;  Rulke,  &  Rau,  1997).  Strategy  formulation  training 
may  have  presented  subjects  with  unique  information  that  could  not  be  as  easily  assimilated 
as  that  presented  in  the  coordination  training  condition  in  the  time  provided.  A  pre-training 
assessment  could  have  provided  some  important  insights  into  the  subject's  of  competency 
and  knowledge  in  these  two  areas  by  illustrating  just  how  much  the  subjects  had  learned 
during  the  training  sessions  rather.  The  resulting  measure  could  be  a  more  accurate 
assessment  of  the  training  conditions  than  either  of  the  two  measures  employed  in  this  study. 

These  procedures  could  be  at  least  partially  responsible  for  the  lack  of  significance 
demonstrated  by  the  training  conditions,  especially  strategy  training,  in  the  analyses.  The 
combined  effects  of  these  unintended  training  opportunities  and  the  lack  of  variation  in 
complexity  levels  across  experimental  missions  could  have  reduced  or  masked  their  potential 
significance  in  a  repeated  measures  regression.  This  is  especially  true  if  the  procedures 
discussed  here  contributed  to  the  two  training  conditions  being  less  distinct  from  each  other 
than  originally  designed.  Future  research  is  required  to  discern  if  this  assessment  accurately 
reflects  potential  confounds  in  the  experimental  design. 

Limitations. 

As  with  any  study  conducted  in  a  laboratory  setting,  there  are  concerns  about  the 
generalizability  of  its  findings.  Given  the  unique  nature  of  MTSs,  every  effort  was  made  to 
ensure  students  were  aware  that  they  were  part  of  a  greater  whole  than  just  their  dyad.  A 
number  of  efforts  were  made  during  training  to  maximize  the  realism  of  the  experimental 
conditions,  such  as  designing  tasks  that  required  all  the  team  members  to  interact  within  and 
between  teams  comprising  the  MTS.  As  evidenced  by  the  number  of  informal  requests  by 
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many  teams  to  return  and  fly  more  missions  as  a  unit,  the  simulation  successfully  held  their 
interest  and  reinforced  perceptions  of  team  identity. 

As  discussed  earlier,  there  were  limitations  in  the  way  data  were  collected  to  test  the 
effects  of  the  experimental  manipulations.  For  example,  using  ratings  based  on  a  single  team 
member’s  responses  to  questions,  while  isolated  from  the  MTS  setting,  as  indices  of 
transition  process  behaviors  may  not  have  been  the  most  effective  means  to  assess  those 
constructs.  Such  ratings  may  not  adequately  reflect  the  actual  interactions  and  planning 
behaviors  underlying  MTS  decisions  before  and  during  the  experimental  missions.  To 
accurately  capture  the  nature  and  quality  of  an  MTS’  transition  processes  may  require 
assessing  these  behaviors  when  they  are  actually  occurring  and  in  conditions  where  all  team 
members  are  interacting  rather  relying  on  a  single  source  summation  of  the  team’s  decisions. 

An  obvious,  and  potentially  serious,  limitation  of  this  research  is  the  subject 
population  comprising  each  experimental  condition.  As  can  be  seen  in  Table  1,  ROTC 
students  were  concentrated  in  the  high  complexity  conditions  and  comprised  the  entire 
sample  base  for  the  coordination  training  condition.  This  allocation  resulted  from  delays  in 
the  availability  of  experimental  conditions  at  the  beginning  of  the  overall  research  project 
and  the  original  design  of  this  study. 

Based  on  discussions  with  the  command  and  cadet  staff  of  the  ROTC  detachments  at 
the  university,  it  was  originally  believed  that  more  than  enough  participants  would  be 
available  from  this  population  to  fill  each  experimental  condition  examined  in  this  study. 
Therefore,  placing  all  subjects  into  the  experimental  conditions  as  they  became  available 
appeared  to  present  no  threat  to  later  analysis.  However,  when  it  later  became  obvious  that 
there  would  not  be  enough  MTSs  from  this  population  to  adequately  test  the  proposed 
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hypotheses,  volunteer  subjects  from  undergraduate  psychology  courses  were  used  to 
complete  the  remaining  conditions.  As  discussed  earlier,  analysis  supported  this  decision  and 
indicated  that  there  was  no  significant  difference  between  the  two  sample  populations’ 
performance  across  the  experimental  conditions.  However,  it  is  conceivable  that  having  all 
the  subjects  in  one  of  the  two  experimental  training  conditions  from  a  single  population 
sample  limited  the  range  of  responses  in  other  areas  as  compared  with  other  samples. 
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Chapter  7 

CONCLUSIONS  AND  IMPLICATIONS 

As  discussed  earlier,  manipulating  environmental  complexity  and  training  conditions 
failed  to  strongly  influence  MTS  processes.  However,  the  results  from  this  study  also 
provided  support  for  reconceptualizing  the  relationships  existing  between  transition,  action, 
and  interpersonal  processes.  Based  on  these  findings,  transition  process  behaviors  provide  a 
critical  link  between  interpersonal  and  action  processes  in  an  MTS.  This  supports  the 
emerging  theory  of  MTS  being  developed  by  Marks  et  al.  (in  preparation)  and  offers  some 
initial  clarifications  of  how  these  team  processes  interrelate  to  enhance  overall  performance. 
This  contributes  to  the  existing  literature  on  team  research  since  there  has  been  little  previous 
examination  of  how  teams  function  and  interact  within  MTSs.  This  study  also  lends 
additional  support  to  research  advocating  the  importance  of  examining  team  behaviors  and 
actions  in  more  dynamic  performance  environments  (e.g.,  Goodwin,  1997;  Mathieu  & 
Goodwin,  1997;  Stout  et  al.,  1990;  Kozlowski  &  Gully,  1996)  that  reflect  how  teams  and 
team  members  function  and  interact  in  multi-team  environments. 

The  pattern  of  results  from  this  study,  including  the  lack  of  significant  findings 
regarding  the  manipulations,  illustrates  the  need  to  better  understand  the  factors  underlying 
MTS  processes  and  performance.  These  findings  point  out  the  importance  of  determining 
and  accurately  sampling  the  environmental  conditions  in  which  MTS  processes  occur.  With 
this  insight,  it  should  be  possible  to  better  test  the  hypothesized  relationships  examined  in 
this  study  and  begin  to  conceptualize  how  MTS  processes  differ  from  individual  teams 
operating  in  less  interdependent  settings. 
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From  an  organizational  manager’s  perspective,  there  is  much  to  be  gained  by 
understanding  that,  just  as  individuals  differ  from  teams,  teams  may  function  quite  differently 
from  MTSs.  Being  able  to  conceptualize  and  predict  the  nature  of  these  differences  can 
greatly  enhance  the  effectiveness  of  an  organization’s  design  and  training  efforts.  Better 
understanding  the  unique  role  transition  process  apparently  plays  in  the  way  an  MTS 
functions  and  performs  is  an  initial  step  in  this  effort.  Clearly  more  descriptive  and 
theoretical  research  is  needed  to  refine  these  differences  and  identify  their  significance  to 
organizational  work  performance  and  MTS  training. 

The  results  of  this  study  also  indicates  that  training  in  coordination  behaviors, 
emphasizing  teamwork  skills,  increases  MTS  performance  by  enhancing  important  action 
process  behaviors.  These  teamwork  skills  may  be  as  important,  or  even  more  so,  at  the  MTS 
level  than  has  been  shown  by  previous  research  examining  single  teams'  processes  (c.g., 
Cannon-Bowers  et  al.,  1995;  Salas  et  al.,  1992;  Stout  et  al.,  1997).  Based  on  the  results 
reported  in  this  paper,  organizational  managers  and  training  designers  should  ensure  their 
training  programs  adequately  address  the  coordination  skills  demanded  by  MTSs  of  the 
individuals  and  teams  comprising  them. 

This  study  builds  on  existing  research  examining  the  impact  of  environmental 
complexity  on  individual  (see  Wood,  1986)  and  team  (e.g.,  Cobb,  1998;  Elliott  et  al.,  1997; 
Foushee,  1987)  performance  by  attempting  to  directly  manipulate  the  complexity  of  a  multi¬ 
team  task  environment.  The  multi-team  nature  of  this  study  provided  a  unique  opportunity  to 
explore  team  performance  and  processes  under  conditions  approaching  the  complexity  found 
in  normal  team  task  environments.  Unfortunately,  unlike  normal  work  environments,  teams 
in  this  study  were  exposed  to  relatively  consistent  levels  of  environmental  complexity  that 
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may  have  prevented  an  adequate  test  of  the  hypotheses  examine  in  this  study.  Increasing  the 
differences  in  the  degree  of  complexity  between  the  conditions  should  provide  a  better  test  of 
the  hypotheses  proposed  in  this  study. 

From  a  descriptive  standpoint,  the  results  of  this  study  still  implied,  despite  its 
limitations,  that  environmental  complexity  seems  to  have  some  differing  effects  on  MTS 
interpersonal  process  behaviors  based  on  the  nature  of  teams’  previous  training  experiences. 
With  additional  research,  it  may  be  possible  to  better  assess  these  relationships  and  inform 
training  designers  how  to  best  select  and  design  differing  training  programs  for  MTSs 
operating  in  complex  task  environments.  Additional  work  is  also  required  to  adequately 
assess  the  significance  of  this  implication  for  MTS  performance  in  applied  settings  and  future 
research. 

Future  Research. 

As  is  the  case  with  many  research  efforts,  this  study  has  probably  raised  more 
questions  than  providing  answers  to  existing  concerns.  Because  this  study  represents  some 
initial  research  into  MTS  processes  and  performance,  there  are  a  number  of  challenging  and 
interesting  directions  for  future  research.  The  most  pressing  need  is  for  additional  research  to 
empirically  clarify  theoretical  conceptualizations  and  relationships.  This  is  needed  to  better 
understand  how  MTSs  may  differ  from  smaller,  individual  teams  and  how  these  differences 
may  impact  experimental  designs  and  organizational  approaches  to  MTS  design. 

One  obvious  area  where  future  research  could  build  on  the  findings  of  this  study 
concerns  refining  the  techniques  used  here  to  manipulate  and  assess  complexity  conditions. 
Based  on  previous  research  (cf.,  Cobb,  1998;  Foushee,  1987;  Swezey  &  Salas,  1992),  there 
may  be  much  to  be  gained  by  increasing  the  variance  between  the  levels  of  environmental 
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complexity  in  which  teams  operate.  Such  a  design  should  enhance  researchers'  ability  to 
detect  and  assess  the  effects  of  differing  degrees  of  environmental  complexity  on  MTS 
processes  and  performance.  An  interesting  aspect  of  this  effort  for  future  research  is  to 
examine  the  varying  tolerances  teams  have  for  increasing  complexity  over  time.  A  better 
understanding  of  how  these  thresholds  originate  and  are  maintained  (or  altered)  within  the 
MTS  would  greatly  enhance  future  examinations  of  the  effects  of  environmental  complexity 
and  identification  of  the  techniques  best  suited  to  control  them. 

Based  on  the  results  from  this  study,  future  research  should  consider  using  more 
finely  grained  assessments  of  MTS  process  behaviors  to  adequately  test  theorized 
relationships  between  constructs.  One  approach  that  should  be  considered  was  developed  by 
Fowlkes  et  al.  (1995)  and  focuses  on  how  teams  react  to  particular  events,  especially 
unexpected  events,  during  their  task  performance.  The  Targeted  Acceptable  Responses  to 
Generated  Events  or  Tasks  (TARGETs)  methodology  allows  an  assessment  of  how  well 
teams  confront  dynamic  circumstances  (Fowlkes  et  al.).  This  technique  has  been  shown  to 
reduce  or  overcome  such  traditional  rating  errors  as  halo  effects,  while  remaining  sensitive  to 
the  fact  that  different  teams  may  handle  the  exact  same  situation  using  very  different 
performance  strategies  (Dwyer  et  al.,  1997;  Dwyer,  Oser,  Salas,  &  Fowlkes,  1999).  While 
these  techniques  should  still  be  augmented  by  the  continued  use  of  behaviorally  based  SME 
ratings  (see  Dickinson  &  McIntyre,  1997),  the  TARGETS  approach  can  provide  a  richer 
assessment  of  team  dynamics  within  the  MTS  than  might  otherwise  be  possible. 

Based  on  the  results  of  this  study,  interviewing  individual  team  leaders  to  assess  the 
quality  of  the  MTS’  transition  processes  may  not  be  able  to  fully  capture  the  richness  of  this 
important  construct.  One  technique  that  could  be  employed  by  future  research  would  be  to 
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examine  these  behaviors  as  they  occur  during  the  pre-task  planning.  Video  taping  these 
planning  sessions  would  not  only  provide  a  record  for  later  comparisons  and  analysis,  but 
could  also  enhance  evaluations  of  emerging  team  leadership,  conflict  management,  and  the 
effectiveness  of  team  training  within  the  MTS. 

While  it  could  be  difficult  at  times  to  discriminate  activities  that  provide  evidence  for 
effective  transition  behaviors  and  those  more  aligned  to  action  or  interpersonal  process,  the 
resulting  data  base  could  give  a  better  researchers  a  richer  means  to  assess  the  quality  of 
MTS’  transition  processes  than  relying  on  one  subject’s  summation  of  the  teams’  intentions. 
To  enhance  the  richness  of  the  interview,  future  researchers  employing  the  simulation  used  in 
this  study  should  also  consider  conducting  it  in  the  presence  of  the  entire  MTS.  Using 
videotaped  recordings,  frequency  measures  of  the  MTS  member’s  reactions  to  and 
corrections  of  the  mission  commander’s  responses  could  enhance  the  quality  of  the 
assessment  and  the  data  available  for  later  analysis. 

Finally,  subjective  ratings  should  be  conducted  and  analyzed  at  both  the  team  and 
MTS  level.  Based  on  the  results  of  this  study,  there  may  be  large  differences  between  the 
processes  demonstrated  by  individual  teams  versus  those  exhibited  by  the  MTS.  For 
example,  both  individual  teams  may  be  demonstrating  a  high  level  of  team  process  as  they 
accomplish  their  assigned  objectives,  yet  not  interacting  at  the  MTS  level  to  any  significant 
degree. 

In  this  study,  it  was  possible,  despite  embedded  penalties  for  doing  so,  for  the  two 
dyads  to  split  apart  at  the  start  of  the  mission  and  not  rejoin  at  any  point  during  the  mission 
nor  coordinate  any  of  their  activities.  If  both  dyads  performed  and  interacted  well,  even  in 
self-imposed  isolation  for  each  other,  the  aggregated  SME  ratings  would  not  be  indicative  of 
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the  quality  (or  lack  of  it)  of  the  larger  MTS’  processes.  Additional  recordings  of  the 
experimental  session  could  be  used  to  minimize  halo  effects  resulting  from  previous  ratings 
at  the  individual  and  dyad  level. 

Some  consideration  should  also  be  given  by  future  research  using  this  simulation  to 
conducting  a  separate  analysis  of  the  two  dyads,  i.e.  wing  and  lead  aircraft.  Researchers 
could  closely  examine  how  the  behaviors  and  performance  of  each  of  these  two  subteams 
separately  affects  MTS  performance.  This  analysis  could  highlight  significant  differences 
between  the  two  dyads  and  possibly  provide  findings  that  might  go  otherwise  undetected. 
Conclusions. 

This  research  illustrates  the  importance  of  transition  processes  in  MTS  functioning 
and  performance  and  thus  contributes  to  a  new  and  growing  body  of  research  expanding  the 
team  literature  into  multi-team  environments.  As  organizations  continue  to  increase  their 
reliance  on  teams  to  accomplish  their  goals  and  objectives,  more  and  more  teams  will  find 
themselves  in  environments  where  their  success  depends  on  how  well  they  orchestrate  their 
collective  actions  within  a  larger  structure,  i.e.  the  MTS.  The  current  research  also  implied 
that  training  and  environmental  complexity  may  potentially  influence  how  well  MTSs 
perform  at  higher  levels  of  complexity.  However,  as  noted  earlier,  this  study  raised  many 
questions  that  require  further  research  to  clarify  and  understand  the  external  and  internal 
factors  impacting  MTS  process  and  performance. 
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NOTES 


1  This  is  the  lowest  rank  for  officers  in  the  US  Air  Force,  while  Master  Sergeant  is  a  senior  ranking 
Noncomissioned  Officer  (NCO)  among  enlisted  personnel. 

Although  the  teams  would  be  given  limited  training  and  resources  for  conducting  air-to-air  combat,  their 
assigned  tasks  in  each  experimental  mission  was  to  identity  and  destroy  selected  ground  targets. 

CRM  emerged  from  civilian  and  military  efforts  to  reduce  flying  errors  and  aviation  accidents  by  improving 
team  process  early  in  a  team’s  formation  (Cavanagh  &  Williams,  1987;  Foushee,  1987;  Prince  &  Salas,  1993). 
Air  Crew  Training  (ACT)  refers  to  the  various  military  versions  of  this  type  of  training  (Guzzo  &  Dickson, 
1996).  Due  to  differing  command  relationships,  missions,  and  policies,  there  is  not  presently  a  uniform  version 
of  CRM  embraced  by  all  military  services. 

4  Students  enrolled  in  the  USAF  and  US  Army  ROTC  programs  are  referred  to  as  cadets.  Students  in  the  US 
Navy,  including  the  US  Marine  option,  are  referred  to  as  midshipmen.  These  designations  are  identical  to  those 
used  at  the  military  academies. 

5  An  USAF  ROTC  cadet  flight  is  a  hierarchical  structure  within  the  organization  containing  about  16-28  cadets. 
The  author  of  this  study  is  a  senior  USAF  officer  and  has  over  10  years  of  operational  experience  in  multiple 

weapons  systems  designing  training  mission  simulations.  Additionally,  the  USAF  ROTC  detachment 
commander  was  consulted  for  his  expertise  on  flight  and  combat  operations.  He  is  a  senior  USAF  officer  with 
extensive  operational  and  command  experience  in  three  different  combat  aircraft.  Before  his  present 
assignment,  he  was  the  Vice  Commander  of  the  48th  Fighter  Wing,  Royal  Air  Force  Lakenheath,  England.  This 
unit  flies  our  most  advanced  tactical  fighter,  the  F-15C,  and  routinely  engages  in  allied  operations. 

7  “TAW  is  probably  the  finest  modem  USAF  flight  simulation  on  the  market  to  depict  the  tactical  and 
operational  level  of  modem  airpower”  (Air  University,  Aerospace  Basic  Course,  1998). 
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Table  1.  Experimental  Condition  Assignment 


TRAINING 

CONDITION 

COMPLEXITY 

LEVEL 

TEAM  SUBJECT  POPULATION 
COMPOSITION 

Strategy 

Low 

2  -  ROTC  (1  -  AFROTC;  1  -  NROTC) 

4  -  Psychology 

High 

4  -  ROTC  (3  -  AFROTC;  1  -  NROTC) 

2  -  Psychology 

Coordination 

Low 

6  - ROTC 

High 

6  -  ROTC  (4  -  AFROTC;  1  -  Army  ROTC;  1- 
NROTC) 

Control 

Low 

2  -  ROTC  (1  -  AFROTC;  1  -  NROTC) 

4  -  Psychology 

High 

3  -  ROTC  (2  -  AFROTC;  1  -  Army  ROTC) 

2  -  Psychology 

1  -  Other  (Gaming  Association) 

Note:  Each  team  listed  successfully  completed  all  three  experimental  missions  in  the  time 
allowed. 


Table  2.  Participant  Demographics 

SEX 


Percent 

Valid 

Percent 

Cumulative 

Percent 

Valid 

Male 

68.8 

71.7 

71.7 

Female 

27.1 

28.3 

100.0 

Total 

138 

95.8 

100.0 

Missing 

System 

6 

4.2 

Total 

144 

100.0 

Age 


Percent 

Valid 

Percent 

Cumulative 

Percent 

Valid  18 

22.9 

23.9 

19 

19.4 

44.2 

20 

28 

19.4 

20.3 

64.5 

21 

24 

16.7 

17.4 

81.9 

22 

17 

11.8 

12.3 

94.2 

23 

3 

2.1 

96.4 

24 

3 

2.1 

98.6 

30 

1 

.7 

.7 

99.3 

31 

1 

.7 

.7 

100.0 

Total 

138 

95.8 

100.0 

Missing  System 

6 

4.2 

Total 

144 

100.0 

ethnicity 


IfiWSi 

Percent 

Valid 

Percent 

Cumulative 

Percent 

Valid 

caucasian/white 

114 

79.2 

82.6 

82.6 

African  American/Black 

7 

4.9 

5.1 

87.7 

Hispanic 

6 

4.2 

4.3 

92.0 

Asian  American 

10 

6.9 

7.2 

99.3 

Other 

1 

.7 

.7 

100.0 

Total 

138 

95.8 

100.0 

Missing 

System 

6 

4.2 

Total 

144 

100.0 
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Table  3.  Task  Functions  Critical  to  Performance  by  Position 


Function  Pilot  Weapon 

Specialist 


select  HUD  displays  (e.g.,  air-to-air,  air-to-ground,  navigation)  A  X 


control  pilot  head  movement/views  X  N/A 


full  up  pilot  aids  (radar  displays)  A  X 


interpret  pilot  aids:  radar  display  X  A 


interpret  pilot  aids:  waypoint  display  A  X 


fly/maneuver  the  plane  X  N/A 


control  plane  orientation  in  relation  to  the  horizon  X 


track  waypoints  A  X 


select/cycle  waypoints  A  X 


track  mission  timing  A  X 


control  manual  emcom  settings  A  X 


control  airspeed  X 


control  airbrakes  X  N/A 


control  altitude  X 


control  communication  switchboard  A  X 


control  communication  as  the  spokesperson  _ I _ A _ _ X 


decision  to  communicate  to  other  flight  team  _  I  A  I  X 


create  shoot  lists  A  X 


control  weapon  selection  A  X 


fire  weapons _ _ X 


determine  enemy  position  (e.g.,  distance,  altitude,  and  airspeed)  X  X 


track  #  and  type  of  weapons  remaining  A  X 


detect  and  evade  enemy  radar  lock _ _ X _ 1  X 


detect  and  evade  enemy  attack  (e.g.,  missile  launch,  cannon  fire,  AAA,  X  X 

etc.) _ 


achieve  radar  lock  on  enemy  targets _ | _ X _ _ X 


release  defense  measures  (e.g.,  chaffs,  flares) _ _ A _ _ X 


select  and  cycle  enemy  targets  _  _  1  A _ _ X 


X:  pilot  or  weapons  specialist  is  responsible  for  carrying  out  this  function 
A:  pilot  or  weapons  specialist  should  be  aware 


Table  4.  Targets  Listing 


MISSION 

TARGET 

TYPES: 

assigned 

TARGET  TYPES: 
bonus 

TARGET  TYPES: 
all  possible 

ADDITIONAL 
Targets  to  increase 
COMPLEXITY 

PRACTICE 

MTS 

4/9  T-80  tanks 

3  mobile  AAA 

1  mobile  SAM 
(Crotale) 

Total:  8 

5  T-80  tanks 

2  bedford  trucks 

Total:  7 

6  M-l  tanks  (friendly) 

7  G-6  SPGs 

9  T-80  tanks 

3  mobile  AAA 

5  humvees  (friendly) 

2  mobile  SAMs  (SA-11) 

1  mobile  SAM  (Crotale) 

7  fuel  trucks 

2  bedford  trucks 

4  SU-27s 

Total:  46 

RIDGE 

JUMPER 

4/6  T-80  tanks 

2  fuel  trucks 

2  mobile  SAMs 
(Crotale) 

Total:  8 

2  T-80  Tanks 

5  bedford  trucks 

Total:  7 

6  M-l  tanks  (friendly) 

6  G-6  SPGs 

4  mobile  AAA 

6  T-80  tanks 

4  AH64  Apaches  (friendly) 

1  Boeing  C767  (neutral) 

5  bedford  trucks 

10  humvees  (friendly) 

2  fuel  trucks 

2  mobile  SAMs  (SA-11) 

1  AAA 

4  SPGs 

3  fire  trucks 

Total:  46 

New  Total:  54 

RIVER  RUN 

2  mobile  AAA 

2/4  hovercraft 

4/6  T-80  tanks 

Total:  8 

2  hovercraft 

2  T-80  tanks 

3  bedford  trucks 

Total:  7 

2  mobile  SAMs  (SA-11) 

2  mobile  AAA 

4  hovercraft 

15  M-l  tanks  (friendly) 

4  Humvees  (friendly) 

2  passenger  ferries  (friendly) 

4  MiG-2  Is 

6  T-80  tanks 

3  bedford  trucks 

4  SU-27s 

2  fire  trucks 

2  gun  boats 

4  SPGs 

Total:  46 

New  Total:  54 

CANYON 

SWEEP 

5  mobile  AAA 

3  mobile  SAMs 
(Crotale) 

Total:  8 

7  Bedford  trucks 

Total:  7 

1  Mig-21 

2 IL-76  transports 

5  mobile  AAA 

3  mobile  SAMs  (Crotale) 

6  M-l  tanks  (friendly) 

9  T-80  tanks 

3  MI24  HIND  helicopters 

2  SU-27s 

7  bedford  trucks 

3  Humvees  (friendly) 

2  Limos  (Neutral) 

1  fire  truck  (Neutral) 

2  fuel  trucks 

4  T-80  tanks 

4  SPGs 

Total:  46 


|  New  Total:  54 
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Table  5.  ANOVA  Summary  Table  --  Potential  Mission  Order  and  Timing  Effects 


T1 

T2 

T3 

F  (2, 33) 

Ridge  Jumper 

Mean  =  95.83 

SD  =  72.80 

N  =  12 

Mean  =  134.17 

SD  =  79.25 

N  =  12 

Mean  =  117.50 

SD  =  70.81 

N  =  12 

.88  n.s. 

River  Run 

Mean  =  113.08 

SD  =  50.56 

N  =  13 

Mean  =  112.73 

SD  =  66.04 

N=  11 

Mean  =  105.83 

SD  =  69.73 

N=  12 

.95  n.s. 

Canyon  Sweep 

Mean  =  158.18 

SD  =  60.14 

N  =  1 1 

Mean  =  120.00 

SD  =  78.21 

N  =  13 

Mean  =  110.83 

SD  =  58.69 

N=  12 

.621  n.s. 

3.12  n.s. 

.78  n.s. 

.104  n.s. 

*indicates  p  <  .05 
^indicates  p<  .01 
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Table  6.  Task  Training 


Subject 

Task  Areas 

Basic  Maneuvering 

Communicating  within  teams 

Maintaining  flight 

Climbing  and  Diving 

Turning  aircraft  and  leveling  flight 

Controlling  speed 

Air  braking 

Changing  cockpit  views 

Recovering  from  stalls 

Avoiding  crashing  into  the  ground 

Interpreting  the  heads-up  display  (HUD) 

Determining  flight  altitude 

Determining  air  speed 

Interpreting  the  horizon  display 

Navigating  Flight  Routes 

Following  flight  plans 

Identifying  and  tracking  waypoints 

Identifying  aircraft  types 

Basic  Air-to-Air  (AA)  Combat 

Interpreting  AA  HUD  information  displays 

Identifying  enemy  targets 

Identifying  friendly  aircraft 

Changing  EMCOM  settings 

Releasing  chaffs  and  flares 

Selecting  AA  weapons 

Targeting  enemy  aircraft 

Firing  weapons 

Evading  enemy  missile  lock  and  attack 

Identify  indications  of  successful  attacks  on  enemy 
targets 

Basic  Air-to-Ground  (AG) 
Combat 

Interpreting  AG  HUD  displays 

Identifying  enemy  ground  targets 

Identifying  friendly  ground  forces 

Selecting  AG  weapons 

Targeting  ground  targets 

Firing  weapons 

Identify  indications  of  successful  attacks  on  enemy 
targets 

Practice  Multiteam  Simulation 
(MTS) 

Communicating  between  wing  and  lead  aircraft 

Identifying  and  tracking  “partner”  aircraft 

Identify  role  of  mission  commander 

Refine  previous  flight  and  combat  skills 
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Table  7.  Task  Training  Competencies  Checklist 

Basic  Aircraft  Maneuvering  Competencies: 

Subjects  should  be  able  to: 

1.  climb.  □ 

2.  dive.  □ 

3.  accelerate.  □ 

4.  break.  □ 

5.  turn.  □ 

6.  recover  from  a  stall.  □ 

7.  look  around  (using  the  joystick).  □ 

8.  ascertain  their  altitude.  □ 

9.  identify  their  speed.  □ 

10.  determine  and  control  their  angle  of  ascent/descent.  □ 

Navigating  a  Flight  Route  Competencies. 

Subjects  should  be  able  to: 

1 .  Follow  a  preplanned  flight  path.  □ 

2.  Change  waypoints  (advance/move  back).  □ 

3.  Activate  and  interpret  pilot  aids  displays  □ 

4.  Increase/decrease  range  of  pilot  aids.  □ 

5.  Switch  HUDs.  □ 

6.  Identify  current  HUD  and  interpret  display.  □ 

7.  Change  EMCOM  levels.  □ 

8.  Identify  stealthiness  of  EMCOM  levels.  □ 

Basic  Air-to-Ground  Competencies 

Subjects  should  be  able  to: 

1.  Identify  and  select  enemy  ground  targets.  □ 

a.  identify  their  range  to  target.  □ 

b.  on  the  infrared  display.  □ 

c.  on  the  HUD.  □ 

d.  identify  SAMS.  □ 

e.  identify  targets  added  to  shoot  list.  □ 

2.  Choose  appropriate  weapons.  □ 

3.  Determine  the  number  of  available  weapons.  □ 

4.  Fire  weapons.  □ 

5.  Add  targets  to  shoot  lists.  □ 

6.  Cycle  targets.  □ 

7.  Evade  enemy  missiles  (including  flares  and  chaff).  □ 

Basic  Air-to-Air  Competencies 
Subjects  should  be  able  to: 

1 .  Identify  and  select  enemy  air  targets.  □ 

a.  their  range  to  the  target.  □ 

b.  identify  a  targeted  enemy  plane.  □ 

c.  cycle  targets  O 

2.  Identify  and  select  the  appropriate  weapon  to  be  used.  □ 

3 .  Identify  the  number  of  available  weapons.  □ 

4.  Evade  enemy  missiles  (including  flares  and  chaff).  □ 
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Table  8.  Team  Training  SME  Ratings 


Descriptive  Statistics 
Correlations  (N  =  228) 


TEAMTR1A 

TEAMTR2A 

TEAMTR3A  TEAMTR1B  TEAMTR2B  TEAMTR3B 

TEAMTR1A  Pearson 
Corr 
Sig.  (2- 
tailed) 

TEAMTR2A  Pearson 
Corr 
Sig.  (2- 
tailed) 

TEAMTR3A  Pearson 
Corr 
Sig.  (2- 
tailed) 

1.000 

-.112 

.091 

-.101 

.129 

1.000 

-.167 

.012 

1.000 

TEAMTR1B  Pearson 
Corr 

.650** 

-.171** 

-.111 

1.000 

Sig.  (2- 
tailed) 

.000 

.010 

.096 

• 

TEAMTR2B  Pearson 
Corr 

-.140* 

.766** 

-.208** 

-.116 

1.000 

Sig.  (2- 
tailed) 

.035 

.000 

.002 

.081 

TEAMTR3B  Pearson 
Corr 

-.132* 

-.182** 

.727** 

-.107 

-.160*  1.000 

Sig.  (2- 
tailed) 

.047 

.006 

.000 

.108 

.015 

**  Corr  is  significant  at  the  0.01  level  (2-tailed). 
*  Corr  is  significant  at  the  0.05  level  (2-tailed). 


Key: 

TEAMTR1  A:  Rater  1  assessment  of  coordination  training  knowledge 
TEAMTR2A:  Rater  1  assessment  of  strategy  training  knowledge 
TEAMTR3A:  Rater  1  assessment  of  interpersonal  training  knowledge 
TEAMTR1B:  Rater  2  assessment  of  coordination  training  knowledge 
TEAMTR2B:  Rater  2  assessment  of  strategy  training  knowledge 
TEAMTR3B:  Rater  2  assessment  of  interpersonal  training  knowledge 
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Table  9.  Team  Training  Manipulation  Checks 


Test  Items  — ► 

Coordination 

Strategy 

Training  Condition 

Coordination  Mean 

1.41a 

.65b 

9.54 

Std  Dev 

.69 

.44 

.65 

Strategy  Mean 

.49b 

1.75a 

9.58 

Std  Dev 

.37 

.76 

.71 

Control  Mean 

.76b 

.60b 

9.40 

Std  Dev 

.40 

.43 

.84 

41.71** 

63.46** 

.85  n.s. 

*  indicates  p  <  .01 
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Table  10.  ANOVA  Summary  Table  --  Environmental  Complexity 


Linear  Effects 

df 

Error  df 

Within 

Time 

2 

276 

Between 

Complexity 

1 

138 

Training 

Two-Wav  Interactions 

2 

138 

Within 

Time  x  Complexity 

2 

276 

Time  x  Training 

4 

276 

Between 

Complexity  x  Training 

Three-Wav  Interaction 

2 

138 

Within 

Time  x  Complexity  x 
Training 

^indicates  p  <  .05 
^indicates  p  <  .01 

4 

276 

F 

.678 

.059 

2.817 

.448 

.431 

.519 

2.414  * 
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Table  11.  MTS  MISSION  SCHEDULE 


MTS  ID 


■EHS31 


fTTSTTg 

nTTiTTST 

TTTsrrai 


Training 

Condition 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Coordination 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Strate 


Control 


Control 


Control 


Control 


Control 


Control 


Control 


Control 


Control 


Control 


Control 


Control 


m 

m 

m 

m 


Mission  Order 


RJ,  RR,  CS 


CS,  RR,  RJ 


RR,  RJ,  CS 


RJ,  CS,  RR 


RR,  CS,  RJ 


CS,  RJ,  RR 


RJ,  RR,  CS 


CS,  RR,  RJ 


RR,  RJ,  CS 


RJ,  CS,  RR 


RR,  CS,  RJ 


CS,  RJ,  RR 


RJ,  RR,  CS 


CS,  RR,  RJ 


RR,  CS,  RJ 


RJ,  CS,  RR 


RR,  CS,  RJ 


CS,  RJ,  RR 


RJ,RR,  CS 


CS,  RR,  RJ 


RR,  RJ,  CS 


RJ,  CS,  RR 


RR,  CS,  RJ 


CS,  RJ,  RR 


RJ,  RR,  CS 


CS,  RR,RJ 


RR,  R  J,  CS 


RJ,  CS,  RR 


RR,  RJ,  CS 


RJ,  RR,  CS 


RJ,RR,  CS 


CS,  RR,  RJ 


RR,  RJ,  CS 


RJ,  CS,  RR 


RR,  CS,  RJ 


CS,  RJ,  RR 


Complexity 


*This  team  was  comprised  of  members  of  an  on-campus  gaming  association.  For  purposes 
of  this  study,  they  were  treated  as  a  ROTC  MTS  during  all  analyses. 


130 


Table  12.  Operational  Team  Process  Definitions 


Team  Process 
Dimension 

Conceptual  Definition 

Operational  Definition  for 

ACES  Performance  Environment 

Mission  Analysis 

Interpretation  and  evaluation  of  the 
team’s  mission,  including  identification 
of  the  mission’s  main  tasks  as  well  as  the 
operative  environmental  conditions  and 
team  resources  available  for  mission 
execution. 

Team’s  attempt  to  understand  the  overall 
MTS  mission  and  their  unique 
contribution  to  the  mission.  This  would 
include  ensuring  that  the  entire  team  has  a 
clear  understanding  of  the  team’s  mission, 
the  major  components  of  the  mission,  and 
the  team  resources  that  are  available  for 
the  mission. 

Goal  Specification 

Identification  and  prioritization  of  goals 
and  subgoals  for  mission 
accomplishment. 

Develop  and  assign  overall  mission  goal 
for  MTS.  Develop  and  assign  goals  for 
each  team  in  the  MTS.  Develop  and 
assign  subgoals  that  help  teams 
accomplish  larger  goals 

Strategy  Formulation 
and  Planning 

Formulation  of  strategies  and  courses  of 
action  for  mission  accomplishment. 

a.  GenericPlanning :  Formulation  and 
transmission  of  a  primary  course  of 
action  for  mission  accomplishment. 

b.  Contingency  Planning :  A  priori 
formulation  and  transmission  of 
alternative  plans  and  strategy 
adjustments  in  response  to 
anticipated  changes  in  the 
performance  environment. 

a.  Develop  a  main  plan  to  take  out 
enemy  ground  weapons  without 
getting  killed.  Communicate  this 
plan  to  team  members. 

b.  Consider  factors  that  might  interfere 
with  the  main  plan,  such  as  enemy 
vehicles  or  friendly  casualties,  and 
develop  some  alternative  plans. 

c.  Reactive  Strategic  A  dj ustment:  The 

alteration  of  existing  strategy  or  plan 
in  response  to  unanticipated  changes 
in  the  performance  environment 
and/or  performance  feedback. 

c.  When  team  is  performing  and 

something  unexpected  occurs,  such  as 
a  plane  running  out  of  mavericks  or 
that  enemies  are  just  spotted  to  the 
rear  of  the  aircraft,  plans  should  be 
adjusted  quickly  to  take  into  account 
the  new  information. 

Monitoring  Progress 
Toward  Goals 

Tracking  task  and  goal  progress  toward 
mission  accomplishment;  interpreting 
system  information  in  terms  of  what 
needs  to  be  accomplished  for  goal 
attainment,  transmitting  team  goal 
progress  to  team  members. 

Tracking  and  reporting  team’s  (teams’) 
progress  on  goals  and  subgoals  in  real 
time,  such  as  flight  routes,  enemies  killed, 
friendly  casualties,  and  time. 
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Table  12.  Operational  Team  Process  Definitions  (cont.) 


Systems  Monitoring 

Tracking  team  resources  and 
environmental  conditions  as  they  relate  to 
mission  accomplishment. 

a.  Internal  Systems  Monitoring : 
Tracking  team  resources  such  as 
personnel,  equipment,  and  other 
information  that  is  generated  or 
contained  within  the  team. 

b.  Environmental  Monitoring : 

Tracking  the  environmental 
conditions  relevant  to  the  team. 

a.  Tracking  team  related  factors  such  as 
weapon  availability,  speed,  fuel, 
altitude,  and  radar  information; 
ensuring  that  these  systems  are 
operating  effectively. 

b.  Tracking  aspects  of  the  aircraft 
environment  such  as  terrain  shifts, 
enemy  locations  and  strength,  and 
friendly  forces. 

Team  Monitoring  and 
Backup  Behavior 

Assisting  team  members  to  perform  their 
tasks.  Assistance  may  occur  by  a) 
providing  a  teammate  verbal  feedback  or 
coaching,  b)  by  assisting  a  teammate 
behaviorally  in  carrying  out  actions,  or  c) 
by  assuming  and  completing  a  task  for  a 
teammate.  This  dimension  includes  the 
provision  of  feedback  and  task-related 
support  and  the  seeking  of  help  from 
teammates  when  necessaiy. 

Keeping  an  eye  on  other  teammates  to 
determine  if  and  when  they  need  help. 

When  necessary,  helping  teammates  with 
their  assigned  roles  by  telling  them  what  to 
do  and/or  how  to  do  it.  When  the 
capability  exists,  teams/team  members  may 
step  in  and  carry  out  a  task  for  another 
team/team  member. 

Coordination  Activities 

Orchestrating  the  sequence  and  timing  of 
interdependent  actions. 

Organizing  how  and  when  team  members 
(and  teams)  will  synchronize  actions  that 
require  the  contribution  of  both  pilot  and 
weapon  specialist,  or  that  require  the 
efforts  of  more  than  one  team  in  multiteam 
situations. 

Conflict  Management 

a.  Preemptive  Conflict  Management: 
Establishing  conditions  to  prevent, 
control,  or  guide  team  conflict 
before  it  occurs. 

b.  Reactive  Conflict  Management: 
Working  through  task  and 
interpersonal  disagreements  among 
team  members. 

a.  Team  member(s)  making  statements 
or  offering  opinions  about  task 
related  issues,  the  way  the  team 
functions  together,  or  personal  issues, 
that  are  likely  to  affect  subsequent 
team  conflict. 

b.  When  conflict  with  the  team  (or 
between  teams)  arises,  team 
member(s)  efforts  to  work  through 
the  conflict. 

Motivating  and 
Confidence  Building 

Generating  and  preserving  a  sense  of 
collective  confidence,  motivation,  and 
task  based  cohesion  with  regard  to 
mission  accomplishment. 

Team  members  efforts  to  motivate  each 
other,  influence  the  level  of  confidence  of 
team  members,  and  influence  the  level  of 
task  cohesion  of  team  members  with 
respect  to  the  mission  at  hand: 

Affect  Management 

Regulating  member  emotions  during 
mission  accomplishment,  including  (but 
not  limited  to)  social  cohesion, 
frustration,  and  excitement. 

Team  members’  influence  on  the  positive 
and  negative  emotions  of  other  members. 
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Table  13.  Transition  Process  Measures  Correlation  Matrix 


Correlations 


r  — 

IMA 

IGS 

ISFP 

WEAP 

INTEL 

SITREP 

QSORT 

IMA 

Pearson  Correlation 

1.000 

.779** 

.769** 

.100 

.207* 

.188* 

.255*’ 

Sig.  (2-tailed) 

.000 

.236 

.013 

.024 

.002 

N 

143 

143 

143 

143 

143 

143 

143 

IGS 

Pearson  Correlation 

.779** 

1.000 

.774** 

.159 

.174* 

.215*’ 

Sig.  (2-tailed) 

.000 

.000 

.446 

.058 

.037 

N 

143 

143 

143 

143 

143 

143 

143 

ISFP 

Pearson  Correlation 

.769*1 

.774** 

1.000 

.104 

Sig.  (2-tailed) 

.215 

Slllf  "3 

N 

143 

143 

143 

143 

143 

WEAP 

Pearson  Correlation 

.104 

1.000 

■Kg 

Sig.  (2-tailed) 

.236 

.446 

.215 

wtm 

n 

N 

143 

143 

143 

144 

mMM 

144 

INTEL 

Pearson  Correlation 

.245** 

1.000 

.204* 

.762*’ 

Sig.  (2-tailed) 

■ 

.015 

.003 

.014 

.000 

N 

143 

■Ha 

143 

144 

144 

144 

144 

SITREP 

Pearson  Correlation 

.174* 

.276** 

.371** 

.204* 

1.000 

.789*’ 

Sig.  (2-tailed) 

.037 

.000 

.014 

.000 

N 

3 

143 

143 

144 

144 

144 

144 

QSORT 

Pearson  Correlation 

.255** 

.215** 

.310** 

aaigg 

1.000 

Sig.  (2-tailed) 

N 

143  | 

143 

143 

9 

mmm 

144 

**•  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*•  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Key: 

IMA:  mission  analysis  rating  from  mission  commander  interview 

KGS:  goal  specification  rating  from  mission  commander  interview 

ISFP:  strategy  formulation  and  planning  rating  from  mission  commander  interview 

WEAP:  weapons  option  selection  score 

INTEL:  score  from  intelligence  reports  sort  (partial  q-sort  score) 

SITREP:  score  from  situation  report  sort  (partial  q-sort  score) 

QSORT:  total  score  from  q-sort  exercise 
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Table  14.  Action  and  Interpersonal  Process  Measures 


Mission 

Unrestricted 

Maximum 

Likelihood 

Chi  Squared 

Action  Process 

a  = 

Interpersonal 

Process 

a  = 

T1 

27.39** 

.82 

.83 

T2 

25.09** 

.83 

.90 

T3 

23.76** 

.81 

.89 

22.63** 

.74 

.90 

Overall — ► 

43.89** 

.81 

.88 

**  significant  at  the  0.01  level 


Table  15.  Lead  and  Wing  Team  Process  Ratings  Correlation  Matrix 


BARS  Rating: 

T1 

T2 

T3 

Overall 

Interpersonal 

.208 

.06 

.32 

.20* 

Process 

Behaviors 

N  =  36 

10 

CO 

11 

£ 

N  =  36 

o 

T— < 

II 

z 

Action  Process 

.57** 

.45** 

Behaviors 

CO 

II 

N=  107 

*  indicates  p  <  .05 

*  indicates  p  <  .01 
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Table  16.  MTS  Performance  Scoring 


Description 

Point  Assessment 

Time  Alive 

The  actual  duration  of  each  dyad’s 
mission,  i.e.  how  long  did  the  aircraft 
survive. 

No  points  awarded.  Time 
recorded  for  later  analysis. 
Maximum  time:  15  minutes 

Aircraft  Status 

The  status  of  each  aircraft  at  the 
mission’s  conclusion.  Summed  to 

MTS  team  level. 

Undamaged  =  30  points/aircraft 
Damaged  =  20  points/aircraft 
Destroyed  =  0  points/aircraft 

Range  =  0  to  60 

Target  Types  - 
Assigned 

Indicates  how  many  of  the  assigned 
(i.e.,  primary)  mission  targets  the 

MTS  team  destroyed. 

20  points/assigned  target 

Range  =  0  to  160 

Target  Types  -  Bonus 

Indicates  how  many  of  the  bonus  (i.e., 
secondary)  mission  targets  the  MTS 
team  destroyed. 

10  points/bonus  target 

Range  =  0  to  70 

Penalty  Deduction 

Indicates  how  many  neutral  targets 
the  MTS  team  destroyed.  Points 
deducted  from  total. 

-  40  points/target 

Range  =  -120  to  0 

Total  Mission  Score 

Summation  of  all  MTS  team  scores 
and  deductions. 

Range  =  -120  to  290 

Table  17.  Descriptive  Statistics 


N 

Minimum 

Maximum 

Mean 

Std.  Deviation 

Variance 

ACTION  PROCESS 

mm 

MVjjM 

4.63 

3.1424 

.6896 

.476 

INTERPERSONAL  PROCESS 

KE9 

5.00 

3.1308 

.7767 

.603 

MTS  PERFORMANCE 

108 

0 

280 

118.33 

66.40 

4408.411 

TRANSITION  PROCESS 

108 

-1.27 

1.72 

.2892 

.6790 

.461 

COORDINATION  TRAINING 

108 

0 

1.00 

.3333 

.4736 

.224 

STRATEGY  TRAINING 

108 

0 

1.00 

.3333 

.4736 

.224 

COMPLEX*COORD 

108 

0 

2.00 

.50 

.7673 

.589 

COMPLEX*STRATEGY 

108 

0 

2.00 

.50 

.7673 

.589 

COMPLEXITY 

108 

1 

2 

1.50 

.50 

.252 

137 


Correlation  is  significant  at  the  .05  level  (2-tailed). 
♦Correlation  is  significant  at  the  .01  level  (2-tailed). 


r 


COMXST  Sig.  (2- 

N 

COMXCO  Sig.  (2- 

N 

Pearson 

STRAIT  Sig.  (2- 

N 

Pearson 

COORDT  Sig.  (2- 

N 

Pearson 

TRANSITI  Sig.  (2- 

N 

Pearson 

PERFO  Sig.  (2- 

N 

Pearson 

1NTERP  Sig.  (2- 

N 

Pearson 

ACTION  Sig.  (2- 

N 

Pearson 

COMPL  Sig.  (2- 

N 

Pearson 

Pearson 

.023 

108 

.218* 

1.000 

108 

.218* 

.558 

108 

.000 

.174 

108 

-.057 

« 

Lx  -X  b 
go  o  -x 
M  N  C0 

i 

m  -x  b) 

GO  O  CD 

co  oo  cn 
* 

2  | 

M  00  * 
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O 

2 

*1 
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1 

IPPjjl 

— x  — x  CO 
CO  O  GO 
CO  00  oo 
* 

M  -x  O 

CO  °  5 

CD  C»  ° 

* 

. . . 

.000 

107 

.394** 

cn  -x 

o  o 

4=s  CD  * 

» 

* 

.665 

108 

1.000 

5 

o 

2 

■ 

■ 

o  -±  to 
00  O  s 
O  M  CD 

— X  — X  ^s| 

O  O  00 
CD  ~vi  cn 
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-.027 

.130 

107 

.178 

.198 

107 

.147 

107 

.125 

.000 

107 

1.000 

cn  -x  o 

2  'si  GO 
* 

* 

z 

“ 

rr 

"0 

.546 

108 

kEBI 

.582 
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.028 

O  -X  -X 

cn  o  co 

co  CD  00 

.078 

108 

.125 

108 

.171 
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.000 
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.125 

.174 
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.394** 

u 

rr 

T 

o 
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.183 
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1.000 
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* 

£ 

H 

.000 

108 

1 
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0)0  0 
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J _ 
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MOO 
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J _ 

cn  -x 
o  o 
o  oo  • 

J _ 

.078 

108 

1.000 

.198 
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.170 
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MOO) 
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o 

o 

o 

X 

c 

H 
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4*  -x 

CD  O 

CO  CO  • 

J _ 

.000 

108 

1.000 

cn  -x  Lx 
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o  00  GO 
» 

-  * 

.582 

108 

-.129 

b  -x  ^sj 
cn  o  oo 
oo  -si  cn 

.338 

108 

-.027 

1.000 

108 

-.093 

C 0 
,  H 

5 

S  8 

00  o 

1 

-X 

M  o 

CO  00  ♦ 

J _ 

.000 

108 

1.000 

i 

4x  O 

8  8  8 
i 

CO  -x  M 
MOO 
O)  03  CO 

S 

.777 

108 

.123 

.276 

107 

.028 

-x  -x  o 
O  O 

cd  oo  cn 

-x  “X  o 
CO  O  M 
GO  CD  CO 
» 

~o 

o 
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o 

. 

108 

.000 
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CO  00  o 

J _ 

CD  -x  b 
MOO 

0)  00  o 

J _ 

i 

A  -x  '-tx 

CD  O  CD 

go  oo  cn 

JL _ 

.546 

108 

-.071 

O  -x  4^ 
cn  o  ~x 

CD  -si  GO 

b  -x 

00  O  -s| 

o  oo  cn 

i 

o  -x  o 
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"o 

o 

3 
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Table  18.  Correlation  Matrix 


Table  19.  ANOVA  Summary  Table  --  MTS  Performance 


Linear  Effects 

df 

Error  df 

F 

Within 

Time 

2 

60 

.405 

Between 

Complexity 

1 

30 

1.290 

Training 

Two-Way  Interactions 

2 

30 

1.246 

Within 

Time  x  Complexity 

2 

60 

2.254 

Time  x  Training 

4 

60 

1.188 

Between 

Complexity  x  Training 

Three-Way  Interaction 

2 

30 

1.409 

Within 

Time  x  Complexity 

4 

60 

.879 

x  Training 

*  indicates  p  <  .05 
**indicates  p  <  .01 
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Table  20.  ANOVA  Summary  Table  --  Interpersonal  Process  Behaviors 


Linear  Effects 

df 

Error  df 

F 

Within 

Time 

2 

58 

1.741 

Between 

Complexity 

1 

29 

1.769 

Training 

Two-Wav  Interactions 

2 

29 

.457 

Within 

Time  x  Complexity 

2 

58 

6.785** 

Time  x  Training 

4 

58 

.639 

Between 

Complexity  x  Training 

Three-Wav  Interaction 

2 

29 

.026 

Within 

Time  x  Complexity 
x  Training 

4 

58 

2.772* 

*indicates  p  <  .05 
**indicatesp<  .01 
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Table  21.  ANOVA  Summary  Table  --  Action  Process  Behaviors 


Linear  Effects 

Within 

Time 

Between 

Complexity 

Training 

Two-Way  Interactions 
Within 


Time  x  Complexity 

2 

60 

1.825 

Time  x  Training 

4 

60 

.207 

Between 

Complexity  x  Training 

2 

30 

.769 

Three-Wav  Interaction 
Within 

Time  x  Complexity  4  60  1.637 

x  Training 

*indicates  p  <  .05 
^indicates  p<  .01 


df  Error  df  F 

2  60  1.24 

1  30  .107 

2  30  2.298 
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Table  22.  ANOVA  Summary  Table  --  Transition  Process  Behaviors 


Linear  Effects 

df 

Error  df 

F 

Within 

Time 

2 

60 

1.574 

Between 

Complexity 

1 

30 

.145 

Training 

Two-Wav  Interactions 

2 

30 

.702 

Within 

Time  x  Complexity 

2 

60 

.016 

Time  x  Training 

4 

60 

.663 

Between 

Complexity  x  Training 

Three-Way  Interaction 

2 

30 

.892 

Within 

Time  x  Complexity 

4 

60 

.436 

x  Training 

indicates  p  <  .05 
**  indicates  p<.  01 
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Table  23.  Time  x  Complexity  Interaction 
Dependent  Variable:  MTS  BARS  interpersonal  rating 


TIME 

COMPLEX 

Mean 

Std. 

Deviation 

N 

1 

Low 

3.2315 

.9005 

18 

High 

3.1574 

.6454 

18 

2 

Low 

3.2941 

.7961 

17 

High 

2.9815 

.6515 

18 

3 

Low 

3.4259 

.8805 

18 

High 

2.7037 

.6251 

18 

Total 

Low 

3.3176 

.8487 

53 

High 

2.9475 

.6562 

54 
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Table  24.  Time  x  Complexity  x  Training  Interaction 


Dependent  Variable:  MTS  BARS  Interpersonal  rating 


TRAINING 

TIME 

COMPLEX 

MeanStd.  Deviation 

N 

Coordination 

1 

Low 

3.7500 

1.0891 

6 

High 

3.1667 

.4472 

6 

2 

Low 

3.2667 

.9022 

5 

High 

3.1944 

.4878 

6 

3 

Low 

3.5833 

1.0737 

6 

High 

3.0000 

.3162 

6 

Strategy 

1 

.4830 

6 

High 

3.1111 

.9108 

6 

2 

Low 

3.4444 

.4792 

6 

High 

2.9167 

.9113 

6 

3 

Low 

3.4444 

.6025 

6 

Hiah 

2.6944 

.8460 

6 

Control 

1 

Low 

2.9444 

.9230 

6 

High 

3.1944 

.6184 

6 

2 

Low 

3.1667 

1.0488 

6 

High 

2.8333 

.5375 

6 

3 

Low 

3.2500 

1.0260 

6 

High 

2.4167 

.5553 

6 

Total 

1 

Low 

3.2315 

.9005 

~18 

High 

3.1574 

.6454 

18 

2 

Low 

3.2941 

.7961 

17 

High 

2.9815 

.6515 

18 

3 

Low 

3.4259 

.8805 

18 

High 

2.7037 

.6251 

18 
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Table  25.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TEAM  PROCESS  AND  ENVIRONMENTAL  COMPLEXITY  ON 
MTS  TEAM  PERFORMANCE 


Observations  =108 
Number  of  teams  =  36 
df 

Between  35 
dfwithin  =  72 
dftotal  ~  1 07 


Average  variance  =  2210.476 
Total  variance  =  4408.41 1 


IVs 

p 

R2 

Increment 

F  Incr 

Rtwn 

F  Incr 

W/in 

df 

Interpersonal 

-.117 

.01 

1.93 

(1,68) 

Action 

.393 

.106 

20.44** 

(1,68) 

Transition 

.058 

.003 

.57 

(1,68) 

Complexity 

-.146 

.02 

1.41 

(1,34) 

Complexity  x 
Coordination 
Training 

-.464 

.014 

.95 

(1,30) 

Complexity  x 

Strategy 

Training 

.062 

0.0 

0.0 

(1,30) 

^indicates  p  <  .05 
**indicates  p  <  .01 
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Table  26.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TRANSITION  PROCESS  BEHAVIORS  ON  MTS 
INTERPERSONAL  PROCESS 


Observations  =107 
Number  of  teams  =  36 
df 

Between  35 
dfwithin  =  7 1 

dftotal  106 


Average  variance  =  .532 
Total  variance  =  .603 


R2 

F  Incr 

F  Incr 

TVs 

_ P _ Increment 

Btwn 

W/in 

Transition 

.147  .012 

16.08** 

(1,  70) 

indicates  p  <  .05 
**  indicates  p  <  .01 
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Table  27.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TRANSITION  PROCESS  AND  COORDINATION  TRAINING 
ON  MTS  ACTION  PROCESS 


Observations  =  108 

Number  of  teams  =  36 

df  Between  =  35 
dfwithin  =  72 
dftotal  ~  1 07 


Average  variance  =  .292 
Total  variance  =  .476 


Tvs 

_ 

R2 

Increment 

Finer 

Rtwn 

F  Incr 

W/in 

Transition 

.26 

.066 

14.62* ** 

0,71) 

Coordination 

Training 

.228 

.05 

3.02 

(1,34) 

Complexity  x 
Coordination 
Training 

-.542 

.02 

1.18 

(1,32) 

*indicates  p  <  .05 

**indicates  p  <  .01 


Table  28.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  STRATEGY  TRAINING  AND  ENVIRONMENTAL 
COMPLEXITY  ON  MTS  TRANSITION  PROCESS 


Observations  =108 

Number  of  teams  =  36 

df  Between  —  35 
dfwithin  =  72 
dftotal  =  1 07 

Average  variance  =  .351 
Total  variance  =  .461 


R 

R2 

Increment 

F  Incr 

Rtwn 

Finer 

W/in 

df 

Strategy 

Training 

-.129 

.017 

.78 

(1,34) 

Complexity  x 

Strategy 

Training 

.701 

.039 

1.78 

(1,32) 

“"indicates  p  <  .05 
*  “"indicates  p  <  .01 
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Table  29.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TEAM  PROCESS  ON  MTS  TEAM  PERFORMANCE 
(TRIMMED  MODEL)  _ 


Observations  =108 

Number  of  teams  =  36 

df  Between  ~  35 
dfwithin  =  72 
dftotal  =  107 


Average  variance  =  2210.476 
Total  variance  =  4408.411 


IVs 

_p _ 

R2 

Increment 

F  Incr 

Rtwn 

F  Incr 

W/in 

rif 

Interpersonal 

-.117 

.004 

.76 

0,68) 

Action 

.309 

.101 

19.21** 

(1,  68) 

Transition 

.113 

.012 

2.05 

(1,  68) 

““indicates  p  <  .05 
**  indicates  p  <  .01 
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Table  30.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TEAM  PROCESS  AND  ENVIRONMENTAL  COMPLEXITY  ON 
MTS  TEAM  PERFORMANCE  (TRAINING  SCALE  SCORES) _ 

Observations  =  108 

Number  of  teams  =  36 

df  Between  35 
dfwithin  =  72 
dftotal  =107 

Average  variance  =  2210.476 
Total  variance  =  4408.411 


IVs 

p 

R2 

Increment 

F  Incr 

Rtwn 

F  Incr 

W/in 

df 

Interpersonal 

-.117 

.01 

1.93 

(1,  68) 

Action 

.393 

.106 

20.44* ** 

(1,  68) 

Transition 

.058 

.003 

.57 

(1,  68) 

Complexity 

-.146 

.02 

1.41 

0,34) 

Complexity  x 
Coordination 
Training 

-.169 

.001 

.07 

0,30) 

Complexity  x 

Strategy 

Training 

.026 

0.0 

0.0 

(1,30) 

*  indicates  p  <  .05 

**  indicates  p  <  .01 
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Table  31.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TRANSITION  PROCESS  BEHAVIORS  ON  MTS 
INTERPERSONAL  PROCESS  (TRAINING  SCALE  SCORES) _ 

Observations  =107 


Number  of  teams  =  36 

df  Between  —  35 
dfwithin  =  71 

dftota!  =  1 06 

Average  variance  =  .532 
Total  variance  =  .603 


R2 

Finer 

F  Incr 

TVs 

p 

■SHI 

Transition 

.147 

.022 

16.08** 

(1,70) 

*indicates  p  <  .05 
**  indicates  p  <  .01 
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Table  32.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  TRANSITION  PROCESS  AND  COORDINATION  TRAINING 
ON  MTS  ACTION  PROCESS  (TRAINING  SCALE  SCORES) _ 


Observations  =  108 


Number  of  teams  =  36 


df  Between  35 
dfwithin  =  72 
dftotal=107 


Average  variance  =  .292 
Total  variance  =  .476 


Tvs 

_p _ 

R2 

Increment 

F  Incr 
...  Btwn 

F  Incr 

W/in 

df 

Transition 

.227 

.048 

10.07** 

(1,71) 

Coordination 

Training 

.279 

.073 

4.59* 

(1,34) 

Complexity  x 
Coordination 
Training 

.166 

0.0 

0.0 

(1,32) 

*indicates  p  <  .05 

**indicates  p  <  .01 
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Table  33.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  STRATEGY  TRAINING  AND  ENVIRONMENTAL 
COMPLEXITY  ON  MTS  TRANSITION  PROCESS  (TRAINING  SCALE  SCORES) 

Observations  =  108 

Number  of  teams  =  36 
df 

Between  35 
dfwithin  =  72 
dftotal  ~  107 

Average  variance  =  .351 
Total  variance  =  .461 

R2  F Incr  F Incr 

- (3 - Increment _ Btwn _ W/in _ df _ 

Strategy  -.148  .022  1.01  (1,34) 

Training 

Complexity  x  .078  .003  .13  (1,32) 

Strategy 

Training _ _ _ 

indicates  p  <  .05 
^indicates  p<  .01 
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Table  34.  RETEST  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF  THE 
EFFECTS  OF  TEAM  PROCESS  AND  ENVIRONMENTAL  COMPLEXITY  ON  MTS 
TEAM  PERFORMANCE  (INTERVIEW) _ 


Observations  =  108 

Number  of  teams  = 

36 

df  Between  “  35 
dfwithin  =  72 
dftotal  =  107 

Average  variance  = 

2210.476 

Total  variance  =  4408.411 

R2 

F  Incr 

F  Incr 

JLYs _ p _ 

■■Increment 

W/in 

Interpersonal  -.117 

.01 

1.91 

0, 68) 

Action  .404 

.110 

21.04** 

(1,  68) 

Transition  .025 

.001 

.19 

(1,  68) 

Complexity  -.148 

.021 

1.49 

(1.34) 

Complexity  x  .085 
Strategy 

Training 

0.0 

0.0 

(1,30) 

Complexity  x  -.458 
Coordination 

Training 

.014 

.95 

(1,30) 

*indicates  p  <  .05 

**indicatesp<.01 
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Table  35.  RETEST  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF  THE 
EFFECTS  OF  TRANSITION  PROCESS  BEHAVIORS  AND  ENVIRONMENTAL 
COMPLEXITY  ON  MTS  INTERPERSONAL  PROCESS  BEHAVIORS 
(INTERVIEW)  _ 


Observations  =  107 

Number  of  teams  =  36 

df  Between  =  35 
dfwithin  —  71 

dftotal  106 


Average  variance  =  .532 
Total  variance  =  .603 


R2 

Finer 

Finer 

TVs 

P 

__  Btwn 

W/in 

df 

Transition 

.149 

.022 

16.08** 

(1,70) 

Complexity 

.236 

.054 

2.22 

(1,34) 

*indicates  p  <  .05 
**  indicates  p  <  .01 
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Table  36.  RETEST  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF  THE 
EFFECTS  OF  TRANSITION  PROCESS  BEHAVIORS  AND  COORDINATION 
TRAINING  ON  MTS  ACTION  PROCESS  BEHAVIORS  (INTERVIEW) _ 

Observations  =  108 


Number  of  teams  =  36 

df  Between  —  35 
dfwithin  =  72 
df,otal=107 


Average  variance  =  .292 
Total  variance  =  .476 


TVs 

_P _ 

R2 

Increment 

Finer 

Rtwn 

Finer 
.  W/in 

Hf 

Transition 

.26 

.065 

14.35** 

0.71) 

Coordination 

Training 

.215 

.044 

2.63 

(1,34) 

Complexity  x 
Coordination 
Training 

-.578 

.023 

1.35 

(1,32) 

indicates  p  <  .05 
**  indicates  p  <  .01 
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Table  37.  SUMMARY  OF  REPEATED  MEASURES  REGRESSION  ANALYSIS  OF 
THE  EFFECTS  OF  STRATEGY  TRAINING  AND  ENVIRONMENTAL 
COMPLEXITY  ON  MTS  TRANSITION  PROCESS  BEHAVIORS  (INTERVIEW) 

Observations  =  108 


Number  of  teams  =  36 


df  Between  35 
dfwithin  =  72 
dftotal  =  107 


Average  variance  =  .351 
Total  variance  =  .461 


JYs _ 

__p _ 

R2 

Increment 

F  Incr 
......  Rtvvn 

Finer 

W/in 

df 

Strategy 

Training 

-.121 

.015 

.64 

(1,34) 

Complexity  x 

Strategy 

Training 

.742 

.036 

1.51 

(1,32) 

*indicates  p  <  .05 
**indicatesp<  .01 
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Table  38.  Summary  of  Results 


Proposed  Hypotheses: 

Supported: 

Original 

Model 

Supported: 

Training 

Scale 

Scores 

Supported: 

Interview 

Ratings 

(Transition) 

Hypothesis  1 :  Higher  levels  of  transition  process  behaviors  will  be 
associated  with  higher  levels  of  multi-team  system  (MTS) 
performance. 

YES- 

mediated 

YES- 

mediated 

NO 

Hypothesis  2:  Higher  levels  of  action  process  behaviors  will  be 
associated  with  higher  levels  of  MTS  performance. 

YES 

YES 

YES 

Hypothesis  3:  Higher  levels  of  interpersonal  process  behaviors  will 
be  associated  with  higher  levels  of  MTS  performance. 

NO 

NO 

NO 

Hypothesis  4:  Higher  levels  of  transition  process  behaviors  will 
result  in  increased  levels  of  effective  team  action  process  behaviors. 

YES 

YES 

YES 

Hypothesis  5:  Higher  levels  of  transition  process  behaviors  will 
result  in  increased  levels  of  effective  team  interpersonal  process 
behaviors. 

YES 

YES 

YES 

Hypothesis  6:  Increased  levels  of  environmental  complexity  will 
have  a  direct  negative  impact  on  teams’  performance  in  multi-team 
environments. 

NO 

NO 

NO 

Hypothesis  7:  Teams  receiving  strategy  training  will  exhibit  more 
effective  transition  process  behaviors  than  other  teams. 

NO 

NO 

NO 

Hypothesis  8:  Teams  receiving  coordination  training  will  exhibit 
more  effective  action  process  behaviors  than  other  teams. 

NO 

YES 

NO 

Hypothesis  9:  There  will  be  less  difference  in  the  performance  of 
teams  receiving  strategy  training  versus  those  that  do  not  (i.e., 
control)  at  lower  levels  of  complexity  than  at  higher  levels. 

NO 

NO 

NO 

Hypothesis  10:  There  will  be  less  difference  in  the  performance  of 
teams  receiving  coordination  training  and  those  that  do  not  (i.e., 
control)  at  lower  levels  of  complexity  than  at  higher  levels. 

NO 

NO 

NO 

Hypothesis  1 1 :  Teams  receiving  strategy  training  will  perform  better 
in  more  complex  situations  than  teams  in  the  control  condition. 

NO 

NO 

NO 

Hypothesis  12:  Teams  receiving  coordination  training  will  perform 
better  in  more  complex  situations  than  teams  in  the  control 
condition. 

NO 

NO 

NO 

Additional  effects  identified  during  analysis: 

Action  process  behaviors  mediate  transition  process  effects  on 
performance. 

YES 

YES 

NO 

Action  process  behaviors  mediate  coordination  training  effects 
on  team  performance. 

NO 

YES  • 

NO 

Coordination  training  enhances  interpersonal  process 
behaviors. 

NO 

YES 

NO 
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Appendix  B.  Analysis  of  Potential  Sample  Population  Effects 

A  series  of  ANOVAs  was  accomplished  to  determine  if  significant  differences 
existed  between  the  two  subject  populations  for  any  of  the  four  dependent  variables 
examined  in  this  study  (i.e.,  performance,  interpersonal  process  behaviors,  action  process 
behaviors,  and  transition  process  behaviors).  Due  to  the  fact  that  the  coordination  training 
condition  was  comprised  solely  of  ROTC  student  participants,  it  was  impossible  to  compute 
any  effects  for  any  relationships  involving  training  condition  and  population  type. 

The  results  of  these  analyses  are  summarized  on  Tables  1-4  of  this  appendix.  These 
results  indicated  that  population  type  had  no  significant  effect  on  MTS  performance. 
However,  as  seen  in  Appendix  B,  Tables  3  and  4,  population  type  did  significantly  influence 
action  and  transition  process  behaviors.  Furthermore,  a  significant  two-way  interaction 
existed  between  environmental  complexity  and  population  type  for  transition  population 
behaviors.  The  ROTC  students  seemed  to  do  slightly  better  in  these  areas  than  the 
undergraduate  psychology  students.  However,  it  was  decided  that  these  effects  did  not 
require  additional  controls  within  the  later  analyses  since  the  primary  dependent  variable  in 
the  model,  i.e.  MTS  performance,  was  not  affected  by  population  type. 
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Appendix  B,  Table  1.  ANOVA  SUMMARY  TABLE  -  MTS  PERFORMANCE 


Linear  Effects 

Within 

Time 

Between 

Type 

Two-Way  Interactions 

Within 
Time  x  Type 

Between 

Complexity  x  Type 

Three-Way  Interaction 
Within 

Time  x  Complexity  x  Type 

indicates  p  <  .05 
**  indicates  p  <  .01 


df  Error  F 

df 

2  60  .57 

1  30  1.34 

2  60  1.69 

1  30  .356 

2  60  2.769 
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Appendix  B,  Table  2.  ANOVA  SUMMARY  TABLE  -  INTERPERSONAL  PROCESS 
BEHAVIORS 


Linear  Effects 

Within 

Time 


df  Error  F 

df 


2  58  1.27 


Between 

Type  1  29  .064 

Two-Way  Interactions 
Within 

Timex  Type  2  58  .048 

Between 

Complexity  x  Type  1  29  .749 

Three-Way  Interaction 

Within 

Time  x  Complexity  x  Type  2  58  1.60 

*  indicates  p  <  .05 
^indicates  p <  .01 
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Appendix  B,  Table  3.  ANOVA  SUMMARY  TABLE  -  ACTION  PROCESS 
BEHAVIORS 


Linear  Effects 

Within 

Time 

Between 

Type 

Two-Way  Interactions 

Within 
Time  x  Type 

Between 

Complexity  x  Type 

Three-Way  Interaction 
Within 

Time  x  Complexity  x  Type 

♦indicates  p  <  .05 
**  indicates  p  <  .01 


df  Error  F 

df 


2  60  1.47 

1  30  13.44** 

2  60  .02 

1  30  .20 

2  60  1.94 


Appendix  B,  Table  4.  ANOVA  SUMMARY  TABLE  -  TRANSITION  PROCESS 
BEHAVIORS 


Linear  Effects 

Within 

Time 

Between 

Type 

Two-Way  Interactions 

Within 
Time  x  Type 

Between 

Complexity  x  Type 

Three-Way  Interaction 
Within 

Time  x  Complexity  x  Type 

indicates  p  <  .05 
*  indicates  p<. 01 


df  Error  F 

df 

2  60  1.61 

1  30  6.25* 

2  60  .06 

1  30  4.51* 

2  60  .20 
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Weapon  Specialist  Pilot 
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Appendix  C.  MTS  Laboratory  Layout 


Appendix  D.  Training  Protocol 


Note:  Italicized  type  is  notes  to  the  experimenter.  Plain  type  is  what  should  be  told  to  the 
participants.  It  is  not  necessary  to  follow  the  scripted  parts  word  for  word  although  it  is  important 
that  the  participants  do  receive  this  information.  The  important  point  is  that  participants  attain  the 
competencies  listed  on  the  task  training  competency  checklists. 

I.  In  this  training  segment,  you  will  learn  the  basics  of  how  to  fly  the  F22  simulator.  Some  of 
you  may  have  had  previous  experience  with  flight  simulators  or  even  with  the  simulation 
being  used  in  this  study.  If  you  are  familiar  with  this  simulation,  we  ask  that  you  only  use 
those  functions  demonstrated  in  the  training.  Otherwise,  the  results  of  this  study  could  be 
jeopardized. 

II.  Begin  Navigating  a  Flight  route. 

In  this  mission  you  will  learn  the  basic  operations  of  the  joystick,  throttle,  and  how  to  read 
information  on  the  Heads-up-Display,  or  HUD.  Next,  you  will  learn  how  to  perform  the  following 
maneuvers  with  the  joystick:  1)  climbing  and  diving;  2)  rolling  from  side  to  side;  and  3)  turning  left 
and  right.  You  will  also  learn  how  to  speed  up  and  slow  down  the  plane.  Then,  you  will  learn  how  to 
look  around  in  the  cockpit. 

A.  Click  on  “ Practice  Mission  1  ”  to  begin  the  simulation. 

B.  Pause  the  game  as  it  begins,  and  point  out  the  following  (use  pictures  of  the  HUD  to 

point  these  out) : 

1.  Point  out  the  roles  of  the  Weapons  Specialist  and  the  Pilot: 

The  Pilot  is  responsible  for  maneuvering  the  aircraft,  controlling  its  speed,  and  firing 
weapons. 

The  Weapons  Specialist  is  responsible  for  the  aircraft’s  radar  settings,  choosing  the 
appropriate  weapons,  releasing  chaffs  and  flares,  and  communication  system. 

2.  Altitude  Display,  right  of  the  HUD ’s  center. 

3.  Speed  Display,  just  left  of  the  HUD ’s  center. 

4.  The  Horizon  bars.  These  are  two  long  “L”  shaped  bars.  When  these  are  in  the  middle 
of  the  screen,  the  plane  is  flying  level. 

5.  The  Pitch  Ladder.  These  are  smaller  “L”  shaped  bars  above  and  below  the  Horizon 
bar.  They  have  numbers  to  the  left  of  them.  The  numbers  tell  you  what  angle  the 
plane  is  pointed  up  or  down. 

6.  The  HUD  mode.  At  the  top  of  the  HUD  display  it  currently  reads  “AA  HUD.”  This 
stands  for  Air-to-air  HUD.  In  this  mode  you  can  obtain  information  about,  target  and 
fire  at  enemy  aircraft.  You  can  switch  the  HUD  mode  by  pressing  “HUD  MODE”  on 
the  keyboard.  Press  “HUD  MODE”  now.  Notice  that  it  now  reads  “AG  HUD.” 

This  stands  for  Air-to-ground  HUD.  In  this  mode  you  can  target  and  fire  at  enemy 
ground  targets.  Press  “HUD  MODE”  until  you  see  “NAV  HUD.”  This  stands  for 
Navigation  HUD.  In  this  mode,  you  can  obtain  information  about  how  well  you  are 
following  your  flight  plan. 

7.  The  clock.  There  is  a  clock  at  the  bottom  left  of  the  screen.  In  later  missions,  you 
will  use  this  clock  to  accomplish  certain  goals  within  a  given  amount  of  time.  The 
weapon  specialist  will  have  to  take  responsibility  for  watching  this  clock. 

C.  Release  pause  on  the  game.  Allow  the  participants  to  practice  maneuvering  the  plane. 

D.  Demonstrate  the  following,  using  a  toy  plane  as  visual  aid: 

1 .  Climbing  -  Pull  the  joystick  back.  Notice  that  the  pitch  ladder  moves.  Identify  the 
angle  you  are  flying  at.  Notice  the  altimeter  rising  and  the  speed  falling.  Pull  back 
until  the  pitch  ladder  reads  “80”  for  an  eighty  degree  angle  of  ascent.  Point  out  that 
the  speed  is  rapidly  falling.  Allow  the  participants  to  fly  nearly  straight  up  until  their 
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planes  loose  momentum  and  stall.  If  your  plane  should  stall,  you  need  to  point  the 
nose  down  to  gain  speed.  As  the  plane  tumbles,  it  will  gain  speed,  and  at  around  200 
knots  it  will  gain  maneuverability  again.  Level  the  plane  off. 

2.  Diving  -  Now  push  the  joystick  forward.  Notice  the  pitch  ladder  is  now  displaying 
negative  numbers  for  a  negative  angle  from  level  flight.  Notice  that  the  altimeter  is 
falling  and  the  speed  is  rising. 

3 .  Straighten  the  plane  out  so  that  you  are  flying  level. 

4.  Pull  the  joystick  to  the  right.  Hold  it  to  the  right.  Notice  that  the  plane  is  rolling  but 
is  still  flying  in  the  same  direction.  Straighten  the  plane  out  again.  Simply  pushing 
the  joystick  to  the  left  or  right  will  not  allow  you  to  turn  the  plane. 

5.  Turning  -  To  turn  the  plane  requires  two  steps.  We  will  first  practice  a  right  turn. 
First,  pull  the  joystick  to  the  right  until  the  plane  is  on  its  side  or  perpendicular  to  the 
ground.  Now,  pull  back  on  the  joystick.  Straighten  the  plane  and  turn  to  the  left. 

6.  Speed  Control  -  To  accelerate,  push  the  throttle  all  the  way  forward.  Notice  that  the 
Engine  Power  display  in  the  bottom  right  of  the  HUD  now  reads  “ENG  140%.”  You 
are  now  flying  at  140  percent  of  the  engine’s  capacity,  the  aircraft’s  maximum 
throttle.  This  is  the  F22’s  fastest  setting,  but  flying  at  140%  for  too  long  could  cause 
you  to  run  out  of  fuel  and  crash.  Notice  that  the  speed  is  increasing.  Now,  pull  the 
throttle  all  the  way  back.  Notice  the  Engine  Power  now  reads  “ENG  5 1%.”  This  is 
the  engine’s  lowest  setting.  Notice  that  your  air  speed  is  decreasing.  At  this  setting, 
the  aircraft  will  soon  loose  momentum  and  tumble  towards  the  ground.  The  normal 
setting  is  at  100%  power.  The  F-22  performs  best  when  flying  at  between  400  and 
550  knots  per  hour. 

7.  Braking  -  You’ve  just  seen  how  to  slow  the  plane  down  using  the  throttle,  another 
way  to  quickly  slow  the  plane  down  is  by  using  an  airbrake.  Using  your  pinky 
finger,  push  the  maroon  button  on  the  bottom  side  of  the  joystick  that  is  facing  away 
from  you.  Notice  that  when  you  push  it  “ABK  ON”  appears  at  the  bottom  right  side 
of  the  screen.  Notice  that  the  plane  is  slowing  down.  Now  push  the  button  again. 
Notice  that  the  HUD  now  reads  “ABK  OFF.” 

8.  Looking  around  -  You  can  use  a  switch  on  the  joystick  to  simulate  head  movement. 
You  can  use  this  switch  to  “look  around”  in  the  cockpit  and  out  of  the  plane.  Using 
your  thumb,  move  the  round  gray  knob  on  the  joystick.  Notice  that  its  movement 
parallels  that  of  your  virtual  head.  You  can  push  the  red  button,  next  to  the  gray 
knob,  to  snap  back  to  the  front  view. 

9.  Now,  fly  towards  the  ground.  At  2500  feet,  level  off  and  fly  straight.  Attempt  to  fly 
as  close  as  you  can  to  the  ground.  Make  sure  they  notice  the  “ground proximity 
warning"  and  PULL  UP  on  the  HUD.  When  you  hear  this  warning,  you  should  pull 
up  quickly  to  avoid  crashing.  Notice  that  you  need  to  anticipate  changes  in  the 
altitude  of  the  ground. 

E.  Basic  Maneuvering  Competencies: 

Here,  we  want  to  make  sure  the  participants  have  learned  the  important  task  competencies 
for  this  mission.  The  experimenter  should  ask  them  to  do  the  following.  If  the  participants 
cannot  do  so,  show  them  how  and  quiz  them  again  later. 

1.  Now  I  would  like  you  to  climb  to  25,000  feet. 

2.  Now  dive  to  15,000  feet. 

3.  Accelerate  to  550  knots. 

4.  Slow  down  to  300  knots. 

5.  Look  to  the  left  of  your  plane...and  now  look  straight  ahead. 

Then  ask  them,  “What  is  your...” 

6.  Altitude. 

7.  Speed. 
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Grab  the  joystick  and  put  the  plane  at  a  30  degree  angle. 

8.  Are  you  flying  up  or  down...and  at  what  angle? 

9.  Stall  out  their  plane  and  have  them  recover. 

Now  we  will  build  upon  the  skills  that  you  have  just  developed  and  teach  you  how  to:  1) 
following  a  flight  plan  and  waypoints;  2)  using  the  navigation  HUD  to  follow  a  flight  plan 
and;  3)  identifying  other  aircraft. 

A.  As  the  simulation  begins,  ask  participants  to  push  “SHIFT  and  PILOT  AIDS.  ”  This 
brings  up  two  radar  screens.  The  screen  on  the  bottom  left  shows  your  plane  in  relation 
to  other  planes  in  the  sky.  The  second  screen,  on  the  bottom  right  shows  your  plane  in 
relation  to  the  flight  plan.  Notice  that  on  the  latter  of  these,  there  are  lines.  This  is  the 
path  that  you  are  supposed  to  follow.  Also,  these  lines  have  circles  with  numbers  on 
them.  These  are  waypoints.  You  are  supposed  to  follow  waypoints  in  the  order  given. 
For  example,  you  proceed  from  waypoint  two  to  three  and  then  to  four,  etc.  The  pilot  is 
responsible  for  watching  the  display  on  the  left.  The  Weapons  Specialist  is  responsible 
for  watching  the  display  on  the  right.  The  W.S.  should  routinely  update  the  pilot  on  their 
relation  to  the  flight  route. 

B.  Make  sure  you  are  in  the  NAV  HUD  mode. 

C.  Release  the  pause  on  the  game.  Instruct  the  participants  to  fly  towards  waypoint  two  at 
20,000 feet.  At  waypoint  two,  point  out  how  the  waypoint  carets  on  the  HUD  move  and 
position  so  that  you  can  fly  to  waypoint  three.  Pause  the  game.  When  you  fly  directly 
over  a  waypoint,  your  computer  will  automatically  switch  the  HUD  settings  to  the  next 
waypoint.  However,  in  later  missions,  you  may  not  be  able  to  fly  directly  over  a 
waypoint  due  to  the  need  to  attack  a  target  or  if  you  are  under  attack.  In  these  cases,  you 
can  manually  switch  to  the  next  waypoint  by  pressing  ADVANCE  WAYPOINT.  Point 
out  how  the  carets  and  “WP2  ”  readout  in  the  NA  V HUD  changes.  Also  point  out  how 
the  circle  around  the  next  waypoint  changes  on  the  right  pilot’s  aid.  If  you  goof  and 
press  W  more  than  once  and  accidentally  advance  to  the  wrong  waypoint,  you  can  press 
“WAYPOINT  BACK”  to  move  the  waypoint  back  one.  Have  them  press  WAYPOINT 
BACK  and  watch  the  waypoint  indicators. 

D.  Pause  the  game.  Explain  EMCOM. 

1 .  EMCOM  refers  to  the  plane’s  radar  systems.  There  are  five  EMCOM  levels.  One  is 
the  stealthiest.  It  is  very  hard  for  enemy’s  to  see  you  in  EMCOM  level  one. 

However,  it  is  also  harder  for  you  to  see  enemy’s  and  to  lock  onto  them.  EMCOM 
five  allows  you  to  gather  the  best  information  concerning  enemies  near  you. 
However,  it  is  also  much  easier  for  enemies  to  see  and  to  shoot  at  you. 

2.  EMCOM  can  be  changed  by  pushing  “EMCOM”  and  then  a  number,  1  through  5,  for 
the  desired  EMCOM  level.  Have  them  change  EMCOM  levels.  Release  pause. 

E.  Have  the  subjects  drop  to  2000 feet  and fly  towards  waypoint  three  at  exactly  500  knots. 
When  it  looks  like  the  particpants  have  the  hang  of  it,  end  the  mission. 

F.  At  waypoint  three  the  game  should  end. 

F.  Navigating  a  Flight  Route  Competencies. 

1.  Take  the  stick  and  put  the  participants  off  course.  Turn  off  their  pilot  aids  and 
switch  the  HUD  to  AG.  Fly  towards  waypoint  two. 

2.  If  the  participants  did  not  demonstrate  how  to  do  so  in  finding  their  way  back  to 
waypoint  two,  ask  them  to: 

a.  Increase  the  range  of  the  pilot  aids. 

b.  Decrease  their  range. 

c.  Switch  to  the  Navigation  HUD. 

d.  Advance  the  waypoint. 

e.  Move  the  waypoint  back. 

3.  If  the  participants  are  on  course,  ask  them  how  they  know  that. 
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4.  Ask  them: 

a.  What  is  your  speed? 

b.  What  is  your  altitude? 

c.  What  do  you  look  at  to  find  the  next  waypoint? 

d.  Change  EMCOM  to  the  level  where  enemies  will  have  the  hardest  time 
“seeing”  you. 

e.  Change  EMCOM  level  where  you  can  “see”  the  most  but  enemies  can  also 
“see”  you  best. 

IV.  Begin  Basic  Air-to-Ground. 

In  this  mission  you  will  learn  air-to-ground  combat  tactics  including:  1)  how  to  identify 
enemy  ground  targets;  2)  how  to  target  enemy  ground  vehicles;  3)  how  to  fire  air-to-ground 
weapons;  and  4)  how  to  identify  surface-to-air  weapons,  or  SAMS  and  the  importance  of 
avoiding  them. 

Here,  the  subjects  will  practice  blowing  up  things  on  the  ground.  When  it  appears  that 
they’ve  attained  their  air-to-ground  competencies,  have  them  proceed  to  waypoint four  where 
they  will  receive  a  brief  overview  of  air-to-air  tactics. 

A.  Click  on  “Practice  Mission  4  ”  to  begin  the  simulation. 

B.  Since  your  objective  here  is  to  destroy  ground  targets,  you  must  be  in  the  AG  HUD 
mode.  Switch  to  this  mode  by  pressing  “CHANGE  AG  WEAPONS”  or  the  “HUD 
MODE”  key  until  you  see  “AG  HUD”  at  the  top  of  the  HUD. 

C.  As  the  subjects  approach  the  first  enemy  ground  target,  pause  the  game.  There  are 
several  types  of  enemy  ground  targets.  In  this  mission,  you  will  attack  tanks,  fuel  tanker 
trucks,  and  SAM  sites.  The  tanks  and  fuel  trucks  are  displayed  on  the  HUD  as  crosses. 
There  are  also  SAM  sites.  These  are  surface  to  air  weapons  that  fire  at  aircraft.  They  are 
displayed  in  the  HUD  as  pentagons.  These  are  the  only  type  of  ground  target  that  can 
also  fire  at  you. 

D.  Notice  that  we  are  approaching  a  ground  target  that  has  not  been  targeted.  The  dotted 
lines  indicate  that  this  target  hasn’t  been  added  to  our  shootlist.  Targets  that  aren’t  in  the 
shoot  list  cannot  be  fired  at.  Add  it  to  the  shoot  list  by  pressing  “T,”  just  like  the  AA 
targets.  Identify  the  type  of  ground  target  we  are  approaching  by  looking  at  the  infrared 
display  just  below  the  HUD.  Your  mission  will  be  to  destroy  enemy  ground  targets.  Use 
pictures  of  these  to  show  what  they  look  like  to  the  players.  There  will  also  be  friendly 
troops  on  the  ground.  Point  out  the  friendly  ground  weaponry  on  the  visual  aids.  Note 
that friendly  Ml  tanks  look  similar  to  enemy  T-80s.  Ml ’s  have  flatter  more  elongated 
turrets.  Or,  if  they  can ’t  see  the  difference,  tell  them  to  look for  HUMVEEs.  If 
HUMVEEs  are  with  the  tanks  then  they  are  friendly  forces.  Do  not  fire  at  the  friendly 
forces. 

E.  Now  that  the  plane  is  targeted,  notice  that  a  circle  has  appeared  around  it  on  the  HUD 
display.  When  you  fire  missiles,  they  will  only  lock  onto  this  target.  If  you  have  more 
than  one  target  on  your  screen  and  you  push  “ADD  TARGET”  to  target,  the  computer 
automatically  selects  the  one  closest  to  you.  If  you  do  not  want  to  fire  at  the  target 
because  it  may  be  a  jeep,  you  must  push  “CYCLE  TARGET”  to  cycle. 

F.  The  distance  to  the  ground  target  is  displayed  next  to  a  vertical  bar  near  the  bottom  right 
side  of  the  HUD.  Next  to  this  bar,  the  distance  is  displayed  in  nautical  miles. 

G.  We  want  to  shoot  the  ground  target,  so  we  have  three  choices  of  weapons.  (As  you 
discuss  their  weapons  choices,  have  them  cycle  the  weapons  by  pressing  backspace  so 
that  the  appropriate  weapon  appears  on  the  screen.  Point  out  in  the  HUD  where  the 
weapon ’s  name  and  number  available  are  displayed.)  The  first  is  CANNONs.  This  is 
the  same  weapon  we  discussed  in  the  last  practice  mission.  To  use  CANNONs  you  must 
be  perfectly  lined  up  with  the  target  and  within  two  miles.  These  should  be  used  only  as 
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a  last  resort.  Another  choice  are  AGM  88’s.  These  are  specially  designed  to  lock  onto 
SAM  sites.  Our  other  choice  is  AGM68.  These  are  heat-seeking  missiles.  They  are 
lethal  within  about  10  miles.  Unless  firing  at  SAM  sites,  these  should  be  your  first  choice 
of  weapons. 

H.  Select  the  AGM68.  When  you  are  within  the  optimum  range,  the  words  “SHOOT”  will 
appear  on  the  HUD  and  you  will  hear  “SHOOT  SHOOT.”  To  fire  the  AGM68,  pull  the 
trigger  on  the  joystick.  After  you  fire  at  one  target,  the  computer  will  automatically 
select  the  next  closest  ground  target.  You  should  check  your  infrared  screen  every  time 
you  fire  to  make  sure  that  a  ground  target  objective  is  what  is  targeted.  Every  time  you 
fire,  the  computer  will  automatically  advance  to  the  next  target  in  the  shoot  list. 
Sometimes,  when  a  group  of  tanks  are  right  next  to  each  other,  it  appears  that  you  are 
firing  at  the  same  target.  If  you  pulled  the  trigger,  heard  the  missile  fire,  or  saw  the 
missile  leave  the  plane  then  the  target  in  the  infrared  display  will  not  be  the  same  that  you 
just  fired  at. 

I.  The  AGM  88’s  which  are  used  against  SAM  launchers  and  triple  A,  can  be  fired  the  same 
way  with  one  exception:  the  higher  the  level  of  EMCOM  you  are  in,  the  farther  the 
distance  you  can  shoot  from. 

J.  An  “X”  will  appear  over  a  ground  target  after  it  has  been  fired  at.  Do  not  waste  missiles 
firing  at  the  same  target.  You  only  have  a  limited  number  of  missiles  and  should  be  used 
only  for  your  primary  targets. 

K.  If  they  haven ’t  already  done  so,  ask  them  to  fly  towards  a  SAM  site.  Point  out  the  threat 
rings  of  the  SAM’s  radar  to  the  participants.  Within  these  circles,  the  SAM  can  ‘‘see  ” 
you.  And  if  they  can  see  you,  they  can  shoot  at  you.  When  a  SAM  fires  a  missile  at  them, 
demonstrate  evasive  maneuvering  techniques 

1.  Turning  away  from  the  SAM,  moving  erratically. 

2 .  Launching  chaffs  and flares. 

L.  Allow  participants  to  practice  as  needed. 

M.  Basic  Air-to-Ground  Competencies 

1.  What  weapon  is  currently  selected? 

2.  How  many  do  you  have? 

3.  Where  are  the  enemy  ground  targets  in  relation  to  you? 

4.  Pick  one  of  the  ground  targets  on  the  HUD  and  ask  them  “What  kind  of  target  is 
that  (e.g.,  a  tank,  fueltruck,  etc.)?”  Participants  should  use  their  MFD  (or 
infrared  display)  to  ID  the  target, 

5.  Hit  the  “C”  key  a  number  of  times  to  cycle  through  the  ground  targets.  Ask  them 
where  the  primary  target  is  at.  They  should  look  for  the  target  with  the  circle 
around  it  on  the  HUD  and  pilot  aids. 

6.  Ask  them  to  pick  the  appropriate  weapon  and  to  fire  at  the  target  when  ready.  They 
should  pick  a  AGM  88  if  it  is  a  SAM  site  and  a  AGM  68  if  it  is  anything  else.  They 
should  wait  until  they  here  and  see  ", SHOOT  SHOOT ”  to  fire. 

7.  After  firing,  ask  them  how  they  know  a  missile  is  heading  towards  the  target.  (An 
“X”  will  appear  over  the  target.) 

8.  Tell  them  to  pretend  that  a  missile  has  been  fired  at  them.  Ask  them  to  show  you 
how  they  would  keep  the  missile  from  hitting  them.  (They  should  turn  and  launch 
chaffs  and flares. 
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N.  When  the  subjects  appear  competent  on  air-to-ground  segment,  instruct  them  to  fly  to 
waypoint  four.  At  waypoint four,  pause  the  game  and  explain  that  they  will  now  receive 
some  basic  air-to-air  training.  This  training  is  just  in  case  they  are  attacked  by  an  enemy 
jet  and  need  to  fight  their  way  out.  However,  you  should  emphasize  that  their  main 
mission  is  to  attack  enemy  ground  targets  and  that  they  should  not  waste  time  going  after 
enemy  planes. 

O.  To  fire  at  an  enemy  plane,  you  must  be  in  the  air-to-air  HUD  mode.  Display  the  AA 
HUD  mode  by  pressing  "CHANGE  AA  WEAPONS  ”  or  the  "HUD  MODE  ”  key  until  you 
see  “AA  HUD  ”  at  the  top  of  the  HUD. 

P.  As  the  subjects  approach  the  first  enemy,  (it  comes  up  quickly)  pause  the  game.  Notice 
that  an  aircraft  is  approaching  that  has  not  been  targeted.  Add  it  to  the  shoot  list  by  pressing 
“ADD  TARGET.”  Identify  the  type  of  plane  that  is  approaching.  It  is  a  MIG-21.  Any  plane 
that’s  name  begins  with  SU  or  MIG  is  an  enemy  plane.  These  planes  will  attempt  to  shoot 
you  down.  You  must  then  attempt  to  either  avoid  the  plane  or  to  shoot  them  down.  All  other 
planes  should  not  be  considered  a  threat  and  should  be  ignored. 

A.  Now  that  the  plane  is  targeted,  notice  that  a  circle  has  appeared  around  it  on  the  HUD 
display.  When  you  fire  missiles,  they  will  only  lock  onto  this  plane. 

B.  We  want  to  shoot  down  the  targeted  plane,  so  we  have  two  choices  of  weapons.  One  is 
CANNONs.  These  are  basically  bullets.  In  order  to  hit  a  plane  with  the  CANNONs,  you 
must  be  within  two  miles  and  be  perfectly  lined  up  with  the  target.  Our  other  choice  is 
AIM9xs.  These  are  heat-seeking  missiles.  They  are  lethal  within  about  10  miles.  You 
can  switch  between  these  missiles  by  pressing  the  “CHANGE  AA  WEAPONS”  key.  Do 
that  now  several  times.  Watch  the  HUD  display  to  see  what  weapons  you  have  selected. 
Look  next  to  the  “WEP”  display. 

C.  Select  the  AIM9x.  A  heat  seeking  box  will  appear  on  the  screen.  This  box  will  enclose 
the  circle  representing  the  enemy  aircraft  on  the  HUD  display  when  you  have  “locked 
on”  to  the  target.  When  within  firing  range,  a  circle  will  appear  in  the  middle  of  the 
HUD.  The  larger  the  circle  is,  the  better  the  chances  are  of  hitting  the  primary  target  with 
an  AIM9X.  Also,  when  you  are  within  the  optimum  range,  the  words  “SHOOT”  will 
appear  on  the  HUD  and  you  will  hear  “SHOOT  SHOOT.”  To  fire  the  AIM9x,  pull  the 
trigger  on  the  joystick. 

D.  After  you  fire  a  missile  at  a  plane,  notice  that  an  “X”  will  appear  over  the  circle.  That 
means  that  the  missile  is  on  its  way  to  the  target.  Do  not  fire  any  additional  missiles 
unless  the  “X”  disappears.  If  it  does,  that  means  that  the  plane  has  outmaneuvered  the 
missile.  You  will  have  a  limited  number  of  missiles  in  these  missions  and  should  not 
waste  them. 

E.  Release  pause  on  the  game  and  allow  participants  to  finish  the  simulation. 

F.  Basic  Air-to-Air  Competencies 
Ask  participants  to: 

1.  Point  out  an  enemy  air  target  on  the  HUD  and  the  pilot  aids. 

2.  Point  out  a  targeted  enemy  air  target. 

3.  Which  target  is  your  primary  target? 

4.  Where  are  you  in  relation  to  the  target? 

5.  Identify  the  range  of  the  target. 

6.  Which  weapon  should  be  used  to  fire  at  the  target? 

7.  How  many  missiles  do  you  have  left? 

8.  Are  you  on  course?  If  not,  get  back  on  course. 

9.  What  is  your  speed? 

10.  What  is  your  altitude? 
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V.  Participants  receive  the  team  training  or  control  condition. 

VI.  Participants  see  brief  video,  which  serves  as  an  advanced  organizer  for  the  multi-team 
session. 

VII.  Practice  Mission  5  (MTS  Mission)  In-Basket  Exercise.  Follow  In-Basket  protocol. 

VIII.  Begin  Practice  Mission  5. 

In  this  mission  you  will  fly  a  mission  with  another  live  team.  You  must  work  together  with 
this  other  team,  your  “wing”,  to  achieve  the  mission  objectives. 

A.  Before  the  mission  begins,  present  participants  with  a  briefing  of  their  mission.  In  this 
mission  and  all  following  missions,  it  is  very  important  that  you  follow  the  objectives. 

Do  not  get  caught  up  in  engagements  that  are  not  related  to  your  mission. 

B.  In  this  mission  you  will  have  a  large  number  of  weapons  at  your  disposal.  However,  in 
later  mission  you  won’t  have  as  many  and  you  will  need  to  be  careful  not  to  waste  any  of 
your  weapons. 

C.  Explain  how  the  communications  switchboxes  work. 

1 .  The  participants  will  work  through  the  mission.  During  and  afterwards,  they  will  be 
given  feedback  about  where  they  can  improve  and  what  they  did  well  on. 

VIII.  Following  the  training  mission,  participants  will  be  given  measures  that  will  be  used  to  evaluate 
the  training. 
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Appendix  E.  Team  Training  Scoring  Criteria 

Use  the  following  to  score  the  open-ended  questions  for  the  team  training  manipulation  check  on  page  five  of 
the  Participant’s  Package.  Use  your  judgment  when  comparing  actual  responses  to  those  listed. 

Coordination: 

1  point  for: 

•  Checking  progress  towards  objectives 

•  Monitoring  the  battlefield  and  aircraft  status 

•  Backing  up  teammates 

•  Coordinating  actions 

0.5  point  for: 

Asking:  (1)  How  many  objectives  have  we  met  already? 

(2)  What’s  the  most  important  thing  to  accomplish  right  now? 

(3)  What  are  we  supposed  to  do  next?” 

(4)  What’s  going  on  inside  our  aircraft? 

(5)  What’s  going  on  outside  our  plane? 

(6)  Check  with  teammate  to  make  sure  you  understand  each  other 

(7)  Monitor  teammate’s  responsibilities 

(8)  Coach  teammate 

(9)  Time  joint  actions 

(10)  Give  teammate  relevant  information 

(1 1)  Adjust  my  behavior  to  match  teammate’s  behavior 

Strategy: 

1  point  for: 

•  Identify  what  your  primary  and  secondary  mission  objectives  are 

•  Understand  and  appreciate  the  larger  context  or  battle  in  which  you  are  working 

•  Develop  a  set  of  plans  and  strategies  that  will  help  you  accomplish  your  mission 

•  Change  your  original  plan  during  the  mission,  if  necessary 

0.5  point  for: 

(1)  Knowing  how  to  decide  when  to  abandon  your  initial  and  contingency  plans 

(2)  How  you  will  generate  and  determine  a  new  plan  of  action 

(3)  Considering:  (1)  what  flight  route  or  “flight  plan”  you  will  fly 
(a)  what  your  primary  (or  most  important)  targets  are 

Qd)  what  your  secondary  (or  less  important)  targets  are 

(c)  what  other  allied  forces  are  doing  in  the  mission 

(d)  what  enemy  activities  take  place  in  the  mission 

(e)  what  kind  of  environment  will  our  mission  take  place  in  (e.g.,  terrain)? 

(4)  Generate  alternative  plans  for  accomplishing  the  mission. 


Interpersonal  Skills: 

1  point  for: 

•  Build  members’  sense  of  confidence  and  motivation 

•  Maintain  positive  team  atmosphere 

•  Identify  ways  to  prevent  team  conflict  before  it  occurs 

•  Control  and  reduce  team  conflict  when  it  occurs.” 

0.5  point  for: 

(1)  reassure  your  teammate  that  your  team  can  accomplish  its  mission  successfully 

(2)  motivate  your  teammate  regarding  accomplishing  your  mission. 

(3)  calm-down  your  teammate  if  she  or  he  is  over-excited 

(4)  excite  your  teammate  if  she  or  he  is  too  board  or  quite. 

(5)  realizing  that  the  conflict  has  become  too  severe 

(6)  employing  win-win  solutions. 

(7)  decide  what  to  do  when  conflict  occurs 
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Appendix  F.  Team  Process  Ratings 


Definition: 


Examples: 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


MISSION  ANALYSIS 

Interpretation  and  evaluation  of  the  team’s  mission,  including  identification  of  the  mission’s 
main  tasks  as  well  as  the  operative  environmental  conditions  and  team  resources  available  for 
mission  execution. 

-  Gathering  appropriate  and  relevant  information 

-  Understanding  the  overall  MTS  mission  and  the  team’s 
contributions  to  the  mission 

-  Identifying  the  main  tasks  and  environmental  contingencies  of  the 
mission 

-  Prioritizes  the  mission  objectives  and  required  tasks 

-  Allocating  team  resources  to  accomplish  each  task 

-  Communicating  the  mission  plan  to  all  team  members 


5 


4 


3 


2 


1 


Team  members  fully  understood  their  individual/flight  team’s  roles  and 
task  responsibilities;  they  also  fully  understood  the  individual  or  flight 
team’s  contribution  to  the  overall  mission. 


Team  members  understood  their  individual/flight  team’s  roles  and  task 
responsibilities;  but  did  not  understand  the  individual  or  flight  team’s 
contribution  to  the  overall  mission. 


Team  members  did  not  understand  their  individual/flight  team’s  roles 
and  task  responsibilities;  nor  did  they  understand  the  individual  or  flight 
team’s  contribution  to  the  overall  mission.  They  had  no  idea  what  their 
mission  objectives  were. 
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GOAL  SPECIFICATION 


Definition: 

Examples: 

Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


Identification  and  prioritization  of  goals  and  subgoals  for  mission  accomplishment. 

-  Developing  and  assigning  overall  mission  goals  for  MTS 

-  Developing  and  assigning  goals  for  each  team  in  the  MTS 

-  Developing  and  assigning  subgoals  that  help  teams  accomplish  larger 
goals 


5 


4 


3 


2 


1 


Identified  and  focused  on  their  primary  and  secondary  goals  throughout 
the  mission. 


Identified  and  focused  on  their  primary  and  secondary  goals  throughout 
half  of  the  mission. 


Displayed  no  identification  of  any  goals  throughout  the  mission. 
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Definition: 

Examples: 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


STRATEGY  FORMULATION  &  PLANNING 

Formulation  of  strategies  and  courses  of  action  for  mission  accomplishment.  This  dimension 
includes  generic  planning,  contingency  planning,  and  reactive  strategic  adjustment. 

-  Developing  a  main  plan  to  take  out  enemy  targets  without  getting  killed 

-  Communicating  plans  to  team  members 

-  Considering  factors  (e.g.,  enemy  vehicles,  friendly  casualties)  that  might 
alter  their  mission  plan 

-  Developing  an  alternative  plan  or  specifically  addressing  how  their  mission  plan  and 

actions  will  be  adjusted  to  unexpected  events 

-  Adjusting  team  actions  or  responsibilities  to  adjust  to  unexpected  events  (e.g.,  plane 
running  out  of  mavericks,  enemies  are  just  spotted  to  the  rear  of  the  aircraft). 

-  Recognizes  how  unplanned  reactions  impact  remainder  of  mission  plan  (e.g.,  weapons 
availability,  flight  route,  altitude  and  airspeed  adjustment) 


5 


4 


3 


2 


1 


Team  members  (1)  developed  a  primary  course  of  action  for  mission 
accomplishment  (2)  considered  a  prior  alternatives  to  their  primary 
course  of  action  and  developed  at  least  two  or  three  secondary  courses  of 
action  (3)  were  able  to  detect  and  quickly  adapt/coordinate  their  actions 
to  unexpected  situations  with  appropriate  behavior.  All  team  members 
are  aware  of  and  understand  how  their  individual  task  responsibilities  fit 
into  the  nrimarv  and  secondary  courses  of  action. 


Team  members  (1)  had  some  difficulty  developing  a  primary  course  of 
action  for  mission  accomplishment  (2)  briefly  considered  a  prior 
alternatives  to  their  primary  course  of  action  and  developed  at  least  one 
secondary  course  of  action  (3)  noticed  and  adapted  their  individual  task 
responsibilities,  but  do  not  coordinate  their  actions  within  the  MTS.  All 
team  members  are  aware  of  but  may  not  understand  how  their  individual 
task  responsibilities  fit  into  the  nrimarv  and  secondary  courses  of  action. 


Team  members  (1)  were  unable  to  develop  a  primary  course  of  action 
for  mission  accomplishment  (2)  did  not  consider  a  prior  alternatives  (3) 
failed  to  notice  unexpected  events  and  situations  and  were  unable  to 
adapt/coordinate  their  actions.  All  team  members  are  unaware  how  their 
individual  task  responsibilities  fit  to  accomplish  the  task  at  hand. 
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MONITORING  PROGRESS  TOWARD  GOALS 


Definition: 


Examples: 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


Tracking  task  and  goal  progress  toward  mission  accomplishment;  interpreting  system 
information  in  terms  of  what  needs  to  be  accomplished  for  goal  attainment,  transmitting  team 
goal  progress  to  team  members. 

-  Tracking  team’s  (teams’)  progress  on  goals  and  subgoals  (e.g.,  flight 
routes,  targets  destroyed,  friendly  casualties,  and  time  expenditure) 

-  Reporting  team’s  (teams’)  progress  on  goals  and  subgoals  (e.g.,  flight 
routes,  targets  destroyed,  friendly  casualties,  and  time  expenditure) 


4 


3 


2 


Maintained  awareness  of  and  tracked  their  primary  and  secondary  goals 
progress  throughout  the  mission.  Understood  which  individual  tasks 
and  flight  team  responsibilities  were  necessary  for  goal  attainment. 


Maintained  awareness  of  and  tracked  their  primary  and  secondary  goal 
progress  throughout  half  of  the  mission  for  individualized  tasks  or  flight 
teams.  Did  not  understand  how  individual  tasks  and  flight  team 
responsibilities  fit  into  goal  attainment. 


Displayed  no  awareness  or  tracking  of  any  goal  progress  throughout  the 
mission. 
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SYSTEMS  MONITORING 


Definition: 


Examples: 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


Tracking  team  resources  and  environmental  conditions  as  they  relate  to  mission 
accomplishment.  This  dimension  includes  internal  systems  monitoring  and  environmental 
monitoring. 

-  Tracking  team  related  factors  (e.g.,  weapon  availability,  speed,  fuel, 

altitude,  radar  information)  and  ensure  that  these  systems  are  operating  effectively 

-  Tracking  aspects  of  the  aircraft  environment  (e.g.,  terrain  shifts,  enemy 
locations  and  strength,  friendly  forces) 


4 


3 


2 


Team  members  effectively  monitor  the  flight  system,  each  other’s 
individual  task  responsibilities,  and  any  communication  generated 
within  the  MTS.  They  also  effectively  monitor  the  external 
environment,  location  of  enemy  targets/threats,  friendly  and  neutral 
forces,  air  and  ground  battles,  etc.;  keeping  in  mind  the  overall  MTS 
mission.  Teams  understand  their  individual  roles  and  task 
resnonsibilities  within  this  changing  environment. 


Team  members  monitor  the  flight  system,  and  their  own  individual  task 
responsibilities.  They  may  be  some  communication  generated  within 
the  MTS,  but  they  do  not  attend  to  it.  They  also  monitor  the  external 
environment,  location  of  enemy  targets/threats,  friendly  and  neutral 
forces,  air  and  ground  battles,  etc.  Teams  understand  their  individual 
roles  and  task  responsibilities  within  this  changing  environment. 


Team  members  have  no  idea  how  to  monitor  the  flight  system,  each 
other’s  individual  task  responsibilities,  and  any  communication 
generated  within  the  MTS.  They  also  fail  to  monitor  the  external 
environment,  location  of  enemy  targets/threats,  friendly  and  neutral 
forces,  air  and  ground  battles,  etc.  Team  members  have  no  idea  what 
their  individual  roles  and  task  responsibilities  are  within  this  changing 
environment. 
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TEAM  MONITORING  AND  BACKUP  BEHAVIOR 


Definition: 


Examples: 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


Assisting  team  members  to  perform  their  tasks.  Assistance  may  occur  by  (a)  providing  a 
teammate  verbal  feedback  or  coaching,  (b)  by  assisting  a  teammate  behaviorally  in  carrying 
out  actions,  or  (c)  by  assuming  and  completing  a  task  for  a  teammate.  This  dimension 
includes  the  provision  of  feedback  and  task  related  support  and  the  seeking  of  help  from 
teammates  when  necessary. 

-  Keeping  an  eye  on  other  teammates  to  determine  if  and  when  they  need 
help 

-  Helping  teammates  with  their  assigned  roles  by  telling  them  what  to  do 
and/or  how  to  do  it 


4 


3 


2 


All  team  members  monitor  each  other  in  enacting  the  appropriate  role 
and  task  requirements  to  successfully  complete  the  overall  mission. 


Team  members  are  more  concerned  with  monitoring  whether  they 
themselves  are  enacting  the  appropriate  role  and  task  requirements  to 
successfully  complete  the  overall  mission.  Little,  if  any  attentional 
resources  are  expended  on  what  other  team  members  are  doing. 


Team  members  fail  to  monitor  each  other  in  enacting  the  appropriate 
role  and  task  requirements.  They  really  don’t  even  pay  attention  to 
what  they  are  doing  themselves. 
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COORDINATION  ACTIVITIES 


Definition: 

Examples: 

Complete  skill 

Very  much  skill 

Adequate  skill 

Some  skill 


Hardly  any  skill 


Orchestrating  the  sequence  and  timing  of  interdependent  actions. 

-  Organizing  how  and  when  team  members  (and  teams)  will  synchronize 
actions  that  require  the  contribution  of  both  pilot  and  weapons  specialist 

-  Organizing  how  and  when  team  members  (and  teams)  will  synchronize 
actions  that  require  the  efforts  of  more  than  one  team  in  multi-team  situations 


4 


3 


2 


Maintaining  smooth  coordination  and  synchronization  of  interdependent 
actions  between  individual  roles  and  flight  teams  in  accordance  with  the 
overall  mission. 


Maintaining  a  minimum  level  of  coordination  and  synchronization  of 
interdependent  actions  between  individual  roles  in  accordance  with  the 
overall  mission.  Team  members  are  not  veiy  considered  coordinating 
the  MTS. 


Complete  lack  of  coordination  and  synchronization  of  interdependent 
actions  between  individual  roles  and  flight  teams.  The  flight  mission  is 
very  disorganized  and  no  one  knows  what  is  going  on. 
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CONFLICT  MANAGEMENT 


Definition: 

Examples: 


Complete  skill 

Very  much  skill 

Adequate  skill 

Some  skill 


Hardly  any  skill 


Establishing  conditions  to  prevent,  control,  or  guide  team  conflict  before  it  occurs.  Working 
through  task  and  interpersonal  disagreements  among  team  members. 

-  Making  statements  or  offering  opinions  about  task  related  issues,  the 

way  the  team  functions  together,  or  personal  issues,  that  are  likely  to  affect  subsequent  team 
conflict 

-  Attempting  to  work  through  conflict  when  conflict  with  the  team  (or 
between  teams)  arises 


4 


3 


2 


All  team  members  are  considerate  of  differences;  they  establish  a 
pleasant  and  cooperative  working  environment  while  encouraging  team 
members  to  present  ideas  and  suggestions  regarding  the  overall  mission. 
Team  members  are  able  to  constructively  discuss  problems.  If  conflict 
does  occur,  team  members  are  able  to  manage  and  contain  the 
disagreements  effectively. 


Team  members  are  sometimes  considerate  of  differences;  they  establish 
a  fair  working  environment  between  flight  teams.  Team  members  are 
able  to  discuss  some  problems  and  resolve  most  types  of  conflict.  Some 
team  members  may  just  “stay  out”  of  any  disagreements  which  arise. 


Team  members  are  inconsiderate  of  differences;  they  establish  an 
unpleasant  and  uncooperative  working  environment  regarding  the 
overall  mission.  Team  members  argue  about  problems  in  a  destructive 
manner  and  often  experience  much  conflict.  They  are  completely 
unwilling  to  discuss  the  issue  at  hand  and  have  no  clue  how  to  resolve 
the  disagreement. 
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MOTIVATING  AND  CONFIDENCE  BUILDING 


Definition: 

Examples: 


Complete  skill 


Veiy  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


Generating  and  preserving  a  sense  of  collective  confidence,  motivation,  and  task  based 
cohesion  with  regard  to  mission  accomplishment. 

-  Motivating  each  other 

-  Influencing  the  level  of  task  cohesion  of  team  members  with  respect  to 
the  mission  at  hand 


4 


3 


2 


All  team  members  exhibit  a  strong  sense  of  collective  efficacy  as  well 
as  self  efficacy.  This  attitude  creates  a  positive  attitude  about  the 
overall  mission,  and  members  seek  to  motivate  one  another  through 
reinforcement  and  praise. 


Team  members  exhibit  a  strong  sense  of  self  efficacy,  but  not  much 
collective.  This  self-centered  attitude  allows  one  to  accomplish  his/her 
own  task  successfully,  but  there  is  not  much  encouragement  or 
motivation  between  team  members. 


Team  members  fail  to  exhibit  any  sense  of  efficacy.  This  attitude 
creates  a  negative  attitude  about  the  overall  mission,  since  there  is  a 
complete  lack  of  encouragement  or  motivation  between  team  members. 
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AFFECT  MANAGEMENT 


Definition: 

Examples: 
Complete  skill 

Very  much  skill 

Adequate  skill 

Some  skill 

Hardly  any  skill 


Regulating  member  emotions  during  mission  accomplishment,  including  (but  not  limited  to) 
social  cohesion,  frustration,  and  excitement. 

-  Influencing  the  positive  and  negative  emotions  of  other  members 


5 


4 


3 


2 


1 


While  carrying  out  the  mission  objectives,  team  members  effectively 
extinguished  negative  emotions  and  enhanced  positive  emotions.  They 
were  able  to  regulate  and  maintain  a  solid  sense  of  emotional  stability 
within  the  larger  team. 


While  canying  out  the  mission  objectives,  team  members  extinguished 
their  own  negative  emotions  and  retain  some  positive  emotions.  They 
were  able  to  regulate  and  maintain  a  moderate  level  of  emotional 
stability  within  their  flight  team,  but  not  so  much  the  larger  team.. 


While  carrying  out  the  mission  objectives,  team  members  failed  to 
extinguish  negative  emotions  and  failed  to  enhance  positive  emotions. 
They  were  unable  to  regulate  and  maintain  any  sense  of  emotional 
stability  within  their  flight  team  or  the  larger  team. 
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Appendix  G.  SME  In-Basket  Q-Sort 


Appendix  H.  MTS  Weapons  Options  and  SME  Rankings 
Each  aircraft  will  be  loaded  with: 

-  1760  Cannon  rounds  (multipurpose)  -  optimal  range  =  2NM 

-  Two  (2)  AIM-9M  (air-to-air)  missiles  -  optimal  range  =  10NM 

PLUS  one  of  the  following  options: 

Option  1: 

6  MAVERICKS  (air-to-ground,  hard  targets)  missiles  -  optimal  range  =  1 ONM 
6  HARMS  (high-speed  anti-radiation  missiles)  -  optimal  range  =  15NM 

Option  2: 

8  MAVERICKS  (air-to-ground,  hard  targets)  missiles  -  optimal  range  =  10NM 
4  HARMS  (high-speed  anti-radiation  missiles)  -  optimal  range  =  1 5NM 

Option  3: 

4  MAVERICKS  (air-to-ground,  hard  targets)  missiles  -  optimal  range  =  10NM 
8  HARMS  (high-speed  anti-radiation  missiles)  -  optimal  range  =  15NM 


SME  Rankings: 


Best  Option  =  2  points;  Next  Best  =  1  point;  Least  Effective  =  0 


Ridge  Juniper 

River  Run 

Option  01 

i 

1 

2 

Option  02 

0 

2 

1 

Option  03 

2 

0 

1 
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Appendix  I.  Post  Mission  Planning  Interview 

A.  Given  that  your  mission  brief  specifies  exactly  what  goals  and  objectives  are  required 
of  your  team,  do  you  have  any  questions  about  them? 

B.  What,  in  your  team’s  estimation,  were  the  most  important  pieces  of  information  that 
you  received?  Why  were  they  the  most  important  pieces  of  information  for  your 
planning? 

C.  From  the  information  that  you  were  provided,  what  else  is  going  on  in  this  mission 
that  might  relate  to  your  assignments?  How  is  that  related  to  your  mission? 

D.  How  do  you  plan  on  achieving  the  objectives  you’ve  identified  for  this  mission? 

What  exactly  will  the  lead  and  wing  aircraft  be  doing?  Do  they  have  any  different 
responsibilities?  If  so,  what? 

E.  What  could  go  wrong  with  your  plans?  It  that  happens,  what  will  you  do?  When  will 
you  decide  whether  to  carry  out  that  action(s)  (i.e.,  what  are  your  decision  points)? 

F.  Finally,  tell  me  which  weapons  load  you  selected  for  this  mission.  Why  that  one? 
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Appendix  J.  Interview  Process  Ratings 


Definition: 


Examples: 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


MISSION  ANALYSIS 

Interpretation  and  evaluation  of  the  team’s  mission,  including  identification  of  the  mission’s 
main  tasks  as  well  as  the  operative  environmental  conditions  and  team  resources  available  for 
mission  execution. 

-  Gathering  appropriate  and  relevant  information 

-  Understanding  the  overall  MTS  mission  and  the  team’s 
contributions  to  the  mission 

-  Identifying  the  main  tasks  and  environmental  contingencies  of  the 
mission 

-  Prioritizes  the  mission  objectives  and  required  tasks 

-  Allocating  team  resources  to  accomplish  each  task 

-  Communicating  the  mission  plan  to  all  team  members 


4 


3 


2 


Demonstrates  a  complete  understanding  of  the  team’s  mission,  goals, 
and  resources  beyond  that  clearly  specified  in  the  mission  briefing; 
Accurately  summarizes  the  team’s  relationship  to  other  operations  and 
agents  in  the  task  environment,  as  well  as  identifying  how  these 
operations  and  agents  impact  the  team’s  assigned  mission  and  strategy. 


Demonstrates  a  general  understanding  of  the  team’s  mission,  goals,  and 
resources  beyond  that  clearly  specified  in  the  mission  briefing; 
Recognizes  that  the  team’s  actions  may  affect  other  operations  and 
agents  in  the  task  environment,  as  well  as  identifying  how  many  of  these 
operations  and  agents  impact  the  team’s  assigned  mission  and  strategy. 


Demonstrates  a  complete  lack  of  understanding  of  the  team’s  mission, 
goals,  and  resources  beyond  that  clearly  specified  in  the  mission 
briefing;  Fails  to  recognize  any  relationship  between  the  team’s  actions 
and  other  operations  and  agents  in  the  task  environment;  Fails  to 
perceive  how  these  operations  and  agents  may  impact  the  team’s 
assigned  mission  and  strategy. 
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GOAL  SPECIFICATION 


Definition: 

Examples: 

Complete  skill 


Veiy  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


Identification  and  prioritization  of  goals  and  subgoals  for  mission  accomplishment. 

-  Developing  and  assigning  overall  mission  goals  for  MTS 

-  Developing  and  assigning  goals  for  each  team  in  the  MTS 

-  Developing  and  assigning  subgoals  that  help  teams  accomplish  larger 
goals 


Demonstrates  a  complete  understanding  of  assigned  mission  goals, 
including  clearly  differentiating  between  and  prioritizing  both  primary 
and  secondary  goals  and  recognizing  how  resources  should  be  best 
allocated  to  achieve  them;  If  required  by  the  team’s  mission  plan, 
clearly  identifies  the  primary  and  secondary  goals  assigned  to  each 
aircraft. 


Acknowledged  the  primaiy  and  secondary  goals  specified  in  the  mission 
briefing;  Recognized  the  differences  between  the  two  types  of  goals  and 
described  how  the  goals  would  be  prioritized  during  the  mission; 
Generally  identified  the  goals  assigned  to  the  MTS  and  to  each 
individual  aircraft,  as  required  by  the  team’s  mission  plan; 
Demonstrated  a  general  understanding  of  the  relationship  between 
weapons  selection  and  assigned  goals. 


Failed  to  differentiate  between  or  prioritize  any  primary  and  secondary 
goals;  Failed  to  identify  any  relationship  between  the  weapons  selected 
and  the  assigned  mission  goals. 
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Definition: 


Examples: 


STRATEGY  FORMULATION  &  PLANNING 

Formulation  of  strategies  and  courses  of  action  for  mission  accomplishment.  This  dimension 
includes  generic  planning,  contingency  planning,  and  reactive  strategic  adjustment. 

-  Developing  a  main  plan  to  take  out  enemy  targets  without  getting  killed 

-  Communicating  plans  to  team  members 

-  Considering  factors  (e.g.,  enemy  vehicles,  friendly  casualties)  that  might 
alter  their  mission  plan 

-  Developing  an  alternative  plan  or  specifically  addressing  how  their  mission  plan  and 

actions  will  be  adjusted  to  unexpected  events 

-  Adjusting  team  actions  or  responsibilities  to  adjust  to  unexpected  events  (e.g.,  plane 
running  out  of  mavericks,  enemies  are  just  spotted  to  the  rear  of  the  aircraft).' 

-  Recognizes  how  unplanned  reactions  impact  remainder  of  mission  plan  (e.g.,  weapons 
availability,  flight  route,  altitude  and  airspeed  adjustment) 


Complete  skill 


Very  much  skill 


Adequate  skill 


Some  skill 


Hardly  any  skill 


4 


3 


2 


Clearly  described  a  primary  course  of  action  to  achieve  mission  goals; 
Summarized  significant  events  that  could  adversely  impact  the  team’s 
mission  plan;  Provided  concise  courses  of  action  to  minimize  the  impact 
of  these  events  on  team  goals;  Identified  the  circumstances  under  which 
contingency  plans  would  be  implemented  or  previously  conceived 
actions  and  responsibilities  would  be  reconsidered  and  adjusted. 


Generally  described  a  primary  course  of  action  to  achieve  mission  goals; 
Briefly  identified  alternatives  to  the  primary  course  of  action;  Briefly 
described  what  events  or  circumstances  would  cause  the  primary  plan  to 
be  adjusted  or  abandoned  for  other  options;  Recognized  the  need  to 
adjust  aircraft  responsibilities  in  the  event  preplanned  courses  of  action 
were  abandoned. 


Failed  to  describe  a  coherent  course  of  action  for  mission 
accomplishment;  Did  not  identify  any  significant  events  that  could 
adversely  impact  their  actions  nor  any  alternatives  to  preplanned  actions; 
Failed  to  consider  the  need  to  adjust  mission  responsibilities  between  the 
two  aircraft  due  to  mission  events. 
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Environmental  Complexity 
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Appendix  K.  TEST  OF  HYPOTHESIZED  MTS  MODEL 


Environmental  Complexity 
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Appendix  L.  TEST  OF  HYPOTHESIZED  MTS  MODEL  (TRAINING  SCALE  SCORES) 
I*PUT  PROCESS 


Environmental  Complexity 
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Appendix  M.  RETEST  OF  HYPOTHESIZED  MTS  MODEL  (INTERVIEW) 
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